acceptodds
Under review as a conference paper at ICLR 2027

OpenVA-Bench: Evaluating Open-World Video Anomaly Understanding

Abstract

Open-world video anomaly understanding requires models to recognize anomalies from known categories and make evidence-based inferences about anomalies outside the predefined category set. When available labels fail to describe an event, forcing it into a known category can produce an incorrect yet plausible interpretation. We introduce OpenVA-Bench to examine this risk through prediction accuracy, confidence reliability, and explanation quality. The benchmark integrates 1,330 videos from three datasets and constructs explicit Known–Unknown category splits, with Normal included in the Known set and Unknown Anomaly available as a rejection option. Across 25 Video-LLMs, stronger known-class recognition does not consistently coincide with reliable unknown rejection. Failures concentrate on semantically related category pairs, incorrect predictions can receive high confidence, and coherent explanations can conflict with reference event descriptions. Neither a larger model nor more input frames consistently yields better performance in the evaluated configurations. These findings expose a gap between recognizing known anomalies and providing reliable judgments in open-world applications. OpenVA-Bench supports systematic evaluation of this gap through prediction, confidence, and reasoning.

Primary area:
datasets and benchmarks
Keywords:
Open-Set Video Anomaly Understanding, Video-Language Models, Unknown Anomaly Recognition, Open-Set Recognition, Unknown Rejection, Visual Reasoning, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2CDpOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OpenVA-Bench, a benchmark for evaluating open-world video anomaly understanding in Video-LLMs. It integrates videos from three datasets and evaluates many models across prediction accuracy, confidence reliability, and reasoning quality. The experiments reveal limitations in unknown anomaly rejection, confidence calibration, and explanation consistency, highlighting challenges in open-world anomaly understanding.

Critical strengths

The paper addresses a meaningful problem by investigating the ability of Video-LLMs to recognize anomalies beyond predefined categories. The benchmark provides a unified evaluation framework covering prediction, confidence, and reasoning, offering a more comprehensive assessment than classification accuracy alone.

Critical weaknesses

1. Insufficient discussion of task definition and scope. The paper frames its task as open-world video anomaly understanding, but the evaluation is limited to video-level classification under predefined Known–Unknown category splits. Open-world VAD can involve broader challenges, such as temporal anomaly localization, explainability [1], and dynamic categories [2]. While different interpretations of open-world settings are reasonable, the authors should better discuss these distinctions and clarify the scope and limitations of their task formulation.

2. Insufficient and potentially biased evaluation. The evaluation exclusively considers general-purpose MLLMs, overlooking existing MLLM-based methods specifically developed for VAD/VAU. Since the proposed task does not appear fundamentally incompatible with these methods, including domain-specific baselines is important for a comprehensive evaluation. Furthermore, the fixed uniform frame sampling strategy may miss short anomalies in long videos, potentially underestimating model performance and making it difficult to distinguish sampling failures from limitations in anomaly understanding.

[1] Hawk: learning to understand open-world video anomalies. NeurIPS 2024.

[2] Language-guided open-world video anomaly detection under weak supervision. ICLR 2026.

Questions for authors
  1. Could the authors clarify the scope of their open-world video anomaly understanding task and discuss its relationship to other methods that have different definitions of open-world VAD/VAU?

  2. Why does the evaluation exclude existing MLLM-based VAD/VAU methods, given that the proposed task does not appear fundamentally incompatible with them?

  3. How do the authors ensure that fixed uniform frame sampling does not miss short anomalies in long videos and consequently underestimate model performance?

  4. Is confidence reliability a distinct capability required for open-world VAD, or is it a general property of classification models that is equally relevant in closed-set settings?

  5. The boundary between known and unknown categories is not always well-defined, as different anomaly categories may share overlapping visual characteristics. For example, if a video depicts law enforcement officers arresting an individual, but the known categories include Abuse while excluding Arrest, classifying the event as Abuse could still be reasonable based on the observed physical interactions. However, under the proposed evaluation protocol, such a prediction would be considered incorrect. More generally, how do the authors determine whether an observed event sufficiently matches a known category? Since there may always exist a more specific or appropriate category outside the predefined set.

10 comments
AuthorsOpenReview ↗

Response to Reviewer 2CDp (1/5)

Thank you for requesting this clarification. OpenVA-Bench studies a specific aspect of open-world video anomaly understanding: whether a Video-LLM can recognize events covered by a supplied category set and reject anomalous events whose categories are absent from that set, together with the confidence and explanations accompanying its decisions. We operationalize this question through a controlled open-set protocol. Each model receives video frames and a candidate-category set with textual definitions, and outputs a category decision, self-reported confidence scores, and an explanation. The output space includes the supplied anomaly categories, Normal, and Unknown_Anomaly. Known/unknown status is determined by whether the harmonized event category is included in the candidate set. Thus, “unknown” is defined relative to the available output categories; it does not imply that the underlying event concept was absent from the model’s pretraining data. This scope complements several existing formulations. UBnormal evaluates detection of anomaly types held out from the benchmark’s training set. Open-vocabulary VAD, such as OVVAD, studies detecting and categorizing both seen and unseen anomalies. Language-guided open-world VAD, exemplified by LaGoVAD, studies adapting anomaly detection to user-provided definitions at inference time. These formulations respectively emphasize generalization across anomaly types, recognition with an expanded semantic vocabulary, and changes in what should count as anomalous. OpenVA-Bench focuses on a complementary decision problem: given the available category definitions, does an anomalous event warrant one of those labels, or should the model reject the available anomaly categories? The benchmark measures known-class recognition and unknown-anomaly rejection, and examines confidence reliability, event-description consistency, and logical coherence. Its video-level protocol therefore addresses category decisions and associated explanations, rather than temporal anomaly localization or incremental learning of newly discovered classes. We will clarify this distinction in the task definition and related work, explicitly describing OpenVA-Bench as a controlled open-set evaluation of Video-LLMs motivated by open-world anomaly understanding, and tie our conclusions to this evaluated setting.

AuthorsOpenReview ↗

Response to Reviewer 2CDp (2/5)

Thank you for raising this point. We agree that existing MLLM-based VAD/VAU methods are relevant baselines for the proposed task. We have therefore expanded the evaluation to include VAD-R1, HAWK-7B, and Holmes-VAU, covering all 1,330 benchmark videos, and assessed both their category predictions and generated explanations. VAD-R1 achieves 39.77% OS*, 26.79% UNK, and 32.01% HOS. HAWK-7B achieves 24.92% OS*, 41.34% UNK, and 31.09% HOS. Holmes-VAU achieves 35.23% OS*, 2.77% UNK, and 5.14% HOS. Their Macro-HOS scores are 25.17%, 29.07%, and 3.94%, respectively. These results reveal different recognition and rejection behaviors: VAD-R1 has the highest known-class accuracy among the three, whereas HAWK-7B has the highest unknown rejection. Holmes-VAU identifies some known categories but rarely rejects anomalies outside the candidate set. We also evaluated the explanations using DeepSeek, GPT, and Claude. Under the original DeepSeek judge, the paired EDC/LC scores are 57.98/94.88 for VAD-R1, 16.29/72.35 for HAWK-7B, and 42.02/63.17 for Holmes-VAU, each calculated on 1,329 records with available references. The three judges produce consistent model orderings for both explanation metrics. These additions demonstrate that OpenVA-Bench can evaluate specialized VAD/VAU systems alongside general-purpose Video-LLMs. Table R2. Specialized VAD/VAU baselines on OpenVA-Bench. All scores are percentages; higher is better.

\hline \text{Model} & \mathrm{OS}^{*} & \mathrm{UNK} & \mathrm{HOS} & \text{Macro-HOS} & \mathrm{EDC} & \mathrm{LC} \\\\ \hline \text{VAD-R1} & 39.77 & 26.79 & 32.01 & 25.17 & 57.98 & 94.88 \\\\ \text{HAWK-7B} & 24.92 & 41.34 & 31.09 & 29.07 & 16.29 & 72.35 \\\\ \text{Holmes-VAU} & 35.23 & 2.77 & 5.14 & 3.94 & 42.02 & 63.17 \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer 2CDp (3/5)

Thank you for highlighting the potential effect of temporal sampling. To examine this issue, we added a reference-interval-guided sampling experiment while retaining a total budget of 32 frames per video. For anomalous videos with usable reference intervals, we sample 16 frames uniformly across the full video and allocate another 16 frames across the event intervals. The additional frames are distributed approximately equally across multiple intervals and sampled uniformly within each interval. The selected frames are then presented in chronological order. Normal videos and videos without usable intervals retain uniform 32-frame sampling. This produces event-guided inputs for 741 anomalous videos; the remaining 589 videos comprise 570 normal videos and 19 anomalous videos without usable intervals. We evaluated Doubao-Seed-2.1-Turbo, Kimi-K2.6, and Qwen3-VL-8B-Instruct on the full 1,330-video benchmark. Compared with the original uniform-sampling results, unknown-anomaly rejection increases from 75.06% to 77.37% for Doubao, 58.66% to 62.20% for Kimi, and 39.72% to 43.96% for Qwen. Their HOS scores increase from 69.51 to 71.97, 64.60 to 68.68, and 42.10 to 46.66, respectively. These results are consistent with improved event coverage benefiting classification and rejection. The principal recognition–rejection pattern also remains: under event-guided sampling, Kimi achieves higher known-class accuracy than Doubao (73.82% versus 67.27%), whereas Doubao achieves higher unknown rejection (77.37% versus 62.20%). The HOS ordering of the three models is unchanged. Event-guided sampling raises HOS by at most 4.6 points (Doubao +2.46, Kimi +4.08, Qwen +4.56) and unknown rejection by at most 4.2 points, with the model ordering unchanged. Uniform 32-frame sampling therefore underestimates performance by a bounded amount — at most 4.6 HOS points — and does not alter any model ranking or the recognition–rejection pattern. Table R3. Uniform versus event-guided sampling on all 1,330 videos, using 32 frames per video.

\hline \text{Model} & \text{Sampling} & \mathrm{UNK} & \Delta\mathrm{UNK} & \mathrm{HOS} & \Delta\mathrm{HOS} \\\\ \hline \text{Doubao-Seed-2.1-Turbo} & \text{Uniform} & 75.06 & \text{—} & 69.51 & \text{—} \\\\ & \text{Event-guided} & 77.37 & +2.31 & 71.97 & +2.46 \\\\ \hline \text{Kimi-K2.6} & \text{Uniform} & 58.66 & \text{—} & 64.60 & \text{—} \\\\ & \text{Event-guided} & 62.20 & +3.54 & 68.68 & +4.08 \\\\ \hline \text{Qwen3-VL-8B-Instruct} & \text{Uniform} & 39.72 & \text{—} & 42.10 & \text{—} \\\\ & \text{Event-guided} & 43.96 & +4.24 & 46.66 & +4.56 \\\\ \hline \end{array}$$
Reviewer_4D4rOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OpenVA-Bench, a benchmark for evaluating whether Video-LLMs can recognize known anomaly categories while rejecting anomalous events outside a test-time candidate set. It combines 1,330 videos from three datasets and evaluates 25 models in terms of known-category accuracy, unknown rejection, self-reported confidence, and explanation quality. The experiments reveal several interesting behaviors, including a mismatch between known-category recognition and unknown rejection, and the absence of a monotonic relationship between model size and open-set HOS.

Critical strengths
  1. The paper studies an important and practically relevant problem that is not captured by existing video anomaly benchmarks. Separating known-category recognition from unknown rejection exposes failures that would otherwise be hidden by aggregate accuracy.
  2. The evaluation covers a broad set of 25 Video-LLMs and reports several informative findings. It provides the complete classification prompt, exact metric formulas, model-specific validity counts, reference-file coverage, judge prompts, and important caveats about adapter/decoding differences and judge validity. In particular, separating EDC from LC is conceptually useful.
Critical weaknesses
  1. The meaning of “unknown” is narrower than the paper's open-world framing may suggest. As the authors correctly acknowledge in the limitations, Unknown is defined relative to the test-time candidate set, rather than by absence from model pretraining or by genuine semantic novelty. This is a valid open-set protocol, but it mainly measures whether a model rejects labels excluded from a particular candidate set. Moreover, some categories have overlapping visual definitions, such as Assault, Abuse, and Fighting. A prediction may therefore be visually defensible under an available Known category while still being counted as an unknown-rejection error. Human validation of category applicability would be needed to support the stronger claim that the available labels truly do not apply.
  2. The benchmark and its main conclusions appear strongly dependent on prompt design, but prompt validity is not sufficiently disentangled from visual understanding. The prompt provides detailed category definitions and even names several held-out anomaly examples. The current definition-removal ablation changes multiple prompt components simultaneously, so it does not establish whether the reported failures arise from visual perception, category reasoning, instruction following, or sensitivity to wording. Consequently, findings such as “larger models do not consistently perform better” and “OS∗OS^* and UNK are misaligned” should currently be interpreted as properties of the complete model–prompt–adapter configuration, rather than as conclusions about model understanding alone. Prompt paraphrasing, removing test-unknown examples, and using oracle event descriptions would help disentangle these factors.
Questions for authors
  1. Are the main model rankings and conclusions stable across prompt paraphrases, category orderings, and prompts that do not explicitly name any evaluated Unknown category?
  2. Can the authors add a control using an oracle event description or caption as input? This would help separate failures of visual perception from failures of candidate-set reasoning and instruction following.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 4D4r (1/2)

Thank you for this suggestion. We evaluated three representative models—Doubao-Seed-2.1-Turbo, Kimi-K2.6, and Qwen3-VL-8B-Instruct—on all 1,330 benchmark videos using four prompt variants: a semantic paraphrase (P1), two fixed permutations of the candidate-category order (P2/P3), and a version removing explicit examples of evaluated unknown categories while retaining the general rejection rule (P4). We compared these results with the original prompt (P0).

The HOS ranking remains Doubao > Kimi > Qwen across all five versions. Under P1–P3, HOS ranges from 71.75–74.65 for Doubao, 68.77–71.10 for Kimi, and 42.15–45.35 for Qwen. These results show consistent relative ordering alongside variation in absolute performance.

The P4 results show a consistent decrease in unknown-anomaly rejection while preserving the HOS ranking. Relative to P0, UNK decreases from 75.06% to 65.13% for Doubao, from 58.66% to 43.88% for Kimi, and from 39.72% to 34.87% for Qwen—reductions of 9.93, 14.78, and 4.85 percentage points, respectively. Thus, rejection performance is sensitive to the removal of explicit unknown-category examples, with the same direction of change observed in all three models.

The central finding that stronger known-class recognition does not consistently imply stronger unknown-anomaly rejection also holds across P0–P4. Kimi achieves higher known-class accuracy than Doubao in every version, whereas Doubao consistently achieves higher unknown rejection. The additional experiments therefore support the stability of this finding and the observed three-model HOS ordering, while identifying absolute rejection performance as sensitive to prompt formulation.

\hline \text{Model} & \text{Prompt} & \mathrm{OS}^{*} & \mathrm{UNK} & \mathrm{HOS} \\\\ \hline \text{Doubao-Seed-2.1-Turbo} & \text{P0} & 64.73 & 75.06 & 69.51 \\\\ & \text{P1} & 65.10 & 79.91 & 71.75 \\\\ & \text{P2} & 66.43 & 80.60 & 72.84 \\\\ & \text{P3} & 69.86 & 80.14 & 74.65 \\\\ & \text{P4} & 65.82 & 65.13 & 65.47 \\\\ \hline \text{Kimi-K2.6} & \text{P0} & 71.89 & 58.66 & 64.60 \\\\ & \text{P1} & 75.31 & 63.28 & 68.77 \\\\ & \text{P2} & 77.30 & 65.82 & 71.10 \\\\ & \text{P3} & 78.46 & 62.12 & 69.34 \\\\ & \text{P4} & 75.88 & 43.88 & 55.61 \\\\ \hline \text{Qwen3-VL-8B-Instruct} & \text{P0} & 44.78 & 39.72 & 42.10 \\\\ & \text{P1} & 44.86 & 42.26 & 43.52 \\\\ & \text{P2} & 44.60 & 39.95 & 42.15 \\\\ & \text{P3} & 44.12 & 46.65 & 45.35 \\\\ & \text{P4} & 43.87 & 34.87 & 38.86 \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer 4D4r (2/2)

Thank you for this suggestion. We added an oracle-style event-description control for Doubao-Seed-2.1-Turbo, Kimi-K2.6, and Qwen3-VL-8B-Instruct. In this control, reference event descriptions replace video frames as model inputs. Standalone category labels, video identifiers, and file paths are excluded from the supplied descriptions. The candidate categories, category definitions, and unknown-rejection rule are retained from P0, with the instructions adapted to text input. We compare both conditions on the same 1,329 videos with available reference descriptions, excluding the single video without a reference from both conditions. Outputs without a valid final prediction are counted as errors. All three models achieve substantially higher HOS with reference descriptions. Doubao improves from 69.51 to 82.37, Kimi from 64.60 to 82.77, and Qwen from 42.10 to 75.20, corresponding to gains of 12.86, 18.17, and 33.10 percentage points, respectively. Under the description-input condition, their known-class accuracies reach 85.25%, 86.39%, and 85.36%, while unknown-anomaly rejection reaches 79.68%, 79.45%, and 67.21%, respectively. These results support two complementary observations. First, supplying explicit event information substantially improves classification, indicating that extracting the relevant event evidence from video is an important bottleneck. Second, classification errors persist when models receive reference descriptions, particularly for unknown-anomaly rejection. By replacing video frames with reference event descriptions, this control examines whether models can make correct category decisions when event information is explicitly available. Errors corrected under this condition suggest difficulties in acquiring relevant evidence from the video. Errors that persist despite accurate and sufficient descriptions indicate difficulties in event interpretation, candidate-category matching, or instruction following. This comparison helps locate the sources of failure along the path from visual evidence to the final category decision.

Reviewer_hvSaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OpenVA-Bench, a benchmark for evaluating open-world video anomaly understanding. It combines 1,330 videos from three existing datasets and defines explicit Known–Unknown category splits to evaluate known-class recognition and unknown-anomaly rejection. The benchmark additionally assesses self-reported confidence, agreement between generated explanations and reference descriptions, and internal logical coherence. Experiments across 25 Video-LLM variants reveal gaps between known-class recognition and unknown rejection, overconfident errors, and coherent explanations that disagree with reference descriptions.

Critical strengths

The problem is practically relevant and clearly motivated. When the correct category is absent from the candidate set, models may select a semantically related category and produce a confident, coherent explanation. Investigating this failure mode adds value beyond conventional anomaly recognition accuracy.

The evaluation covers a broad range of models and configurations. The paper evaluates 25 model variants and examines sensitivity to frame count, category count, Known–Unknown splits, and category definitions.

The analysis offers useful diagnostic insights. Distinguishing unknown anomalies misclassified as known anomalies from those classified as normal, while separately examining predictions, confidence, and explanations, helps characterize different failure modes.

Critical weaknesses

The evaluation framework lacks sufficient validation. EDC measures agreement with unverified reference descriptions, while LC measures textual coherence rather than video-grounded correctness.

Known–Unknown category boundaries are ambiguous. The Abuse definition may overlap with Assault, potentially penalizing predictions consistent with the prompt.

Experimental conclusions face confounding and statistical uncertainty. The category-count experiment also changes category composition and evaluation subsets. Small categories and missing confidence intervals further limit conclusions about performance differences.

Missing evidence and category applicability failures are not separated. Sampling 32 frames may omit brief anomalies.

Questions for authors

Table 1 reports EDC/LC scores for Doubao-Seed-2.1-Turbo, whereas Appendix C.3 and Table A5 indicate that this model has no valid reasoning evaluation records. Could you reconcile this discrepancy and clarify the source and sample counts of these scores?

How do you ensure that the Known and Unknown categories are distinguishable under the supplied definitions, particularly Abuse versus Assault? Could a prediction consistent with the prompt’s definition nevertheless be penalized because of the source dataset label? Please provide human validation or results on an unambiguous subset.

How accurate and complete are the reference descriptions, and how reliably does the LLM judge apply the EDC/LC criteria? Could you provide human–judge agreement and cross-judge validation, particularly for the high-LC/low-EDC cases supporting a central finding?

I am open to raising my score if the rebuttal convincingly addresses these concerns through clear explanations and supporting evidence, particularly regarding category boundaries, evidence visibility, and the validity of explanation scoring.

5 comments
AuthorsOpenReview ↗

Response to Reviewer hvSa (1/3)

1.Thank you for identifying this inconsistency. We have rechecked the record-level evaluation files. The archived DeepSeek-V4-Flash evaluations for this model, contain 1,329 valid EDC scores and 1,330 valid LC scores. Matching by model, dataset, and video identifier, with identical explanation text, yields 1,329 valid pairs. EDC uses event_recognition_score, and LC uses logical_coherence_score. Their paired score sums are 2,030 and 2,527; normalization by 2×1,3292\times1,329 gives EDC = 76.37 and LC = 95.07 on the percentage scale. The one LC-only record lacks a reference annotation and is excluded from both paired means. We will correct Appendix C.3 and Table A5 in Appendix D. Including this model gives 32,834 valid pairs across 25 models, with 5,577 pairs (16.99%) receiving LC = 2 and EDC = 0. We will synchronize Section 4.2, Figure 4(b), and Appendix C.4 accordingly. The prediction metrics are unchanged.

AuthorsOpenReview ↗

Response to Reviewer hvSa (2/3)

Thank you for raising this question. We conducted a human applicability audit of the original prompt definitions, focusing on three visually or semantically related category pairs: Abuse–Assault, Theft–Robbery, and Explosion–Fire. Rather than asking annotators to reproduce source dataset labels, we directly assessed whether each observed event was covered by the supplied Known-category definition.

The audit included 108 videos: all 10 Abuse, 11 Assault, 26 Theft, 19 Robbery, and 21 Fire videos, together with 21 randomly sampled Explosion videos. Ten annotators independently reviewed the 32-frame sampled inputs and the corresponding original Known-category definition, with source labels, filenames, model predictions, and other annotators’ responses hidden. We did not supply definitions for the paired Unknown categories, avoiding additional distinctions absent from the original protocol. Each annotator selected “Applicable,” “Not applicable,” or “Undetermined” and provided supporting visual evidence and the relevant definition clause, yielding 1,080 judgments.

The human results support substantial distinguishability under the supplied definitions on the audited examples. For Abuse–Assault, annotators judged the Abuse definition applicable in 87/100 judgments on Abuse videos and not applicable in 103/110 judgments on Assault videos, corresponding to a Known acceptance rate of 87.00% and an Unknown exclusion rate of 93.64%. The respective Undetermined rates were 5.00% and 2.73%. Known acceptance and Unknown exclusion rates were 93.46% and 97.89% for Theft–Robbery, and 100% and 92.86% for Explosion–Fire. These findings indicate that shared actions or visual cues do not generally lead human annotators to treat the paired categories as interchangeable.

We also examined the possibility that source-derived targets penalize predictions consistent with the supplied definitions. After separating “Not applicable” and “Undetermined” responses, the numbers of judgments accepting an Unknown video under the corresponding Known definition were 4/110 (3.64%) for Assault under Abuse, 2/190 (1.05%) for Robbery under Theft, and 15/210 (7.14%) for Fire under Explosion. These nonzero rates identify potential disagreements between source-category membership and definition applicability. We therefore distinguish cross-category acceptance from explicit exclusion and uncertainty, rather than treating every source-label disagreement as evidence of model failure.

As a complementary model control, claude-fable-5-2 achieved Known acceptance rates of 90.00%, 92.31%, and 100%, and Unknown exclusion rates of 90.91%, 100%, and 90.48%, for the three pairs, respectively. The model results are consistent with the overall pattern of the human audit. Together, these results support the distinguishability of the tested Known–Unknown category pairs under the supplied definitions, despite their shared visual features.

AuthorsOpenReview ↗

Response to Reviewer hvSa (3/3)

Thank you for asking us to clarify reference quality and the reliability of the EDC/LC evaluation. The UCF-Crime and MSAD reference descriptions were annotated with direct participation from the authors and collaborators and were previously used in Alert-CLIP. The XD-Violence descriptions were obtained from Holmes-VAU and checked by the authors. We will clarify the annotation sources, human involvement, and processing workflow in the manuscript. We supplemented the original DeepSeek evaluation with GPT and Claude evaluations using the same inputs, scoring criteria, and 0–2 scale. EDC assesses consistency between the reference event description and the model explanation, whereas LC assesses the internal coherence of the explanation. Comparisons match records by model, dataset, and video and verify the corresponding evaluation inputs. For EDC, exact agreement is 84.83% between DeepSeek and GPT, 86.46% between DeepSeek and Claude, and 87.90% between GPT and Claude.For LC, the corresponding exact agreement rates are 89.68%, 81.61%, and 84.38%. We additionally conducted a blinded human evaluation. Two annotators independently scored the explanations before forming the final human scores. Of 400 selected records: 250 representative records, 75 records selected for the original DeepSeek high-LC/low-EDC pattern, and 75 records selected for cross-judge disagreement. We analyze these groups separately to distinguish representative agreement from targeted diagnostic evaluation. On the 250 representative records, human–judge EDC exact agreement is 82.51% for DeepSeek, 78.90% for GPT, and 84.52% for Claude. For LC, exact agreement is 82.11%, 77.87%, and 78.23%. The targeted human evaluation also directly examines the central high-LC/low-EDC finding. Among the 75 selected records, human annotators confirm EDC=0 for 64 records and LC=2 for 68 records, with both conditions confirmed for 59 records (78.67%). Across the full evaluation, the same score combination occurs in 16.99% of DeepSeek-scored records, 12.80% of GPT-scored records, and 9.51% of Claude-scored records. Together, these results support the occurrence of internally coherent explanations that are inconsistent with reference event descriptions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.