acceptodds
Under review as a conference paper at ICLR 2027

Look Inside Each Video: Rethinking How Video Anomaly Detection Is Evaluated

Abstract

Video anomaly detection (VAD) aims to localize anomalous events by identifying when they occur. A common evaluation pools frame-level anomaly scores across all test videos and measures whether anomalous frames receive higher scores than normal frames (Micro-AUROC). This pooling creates comparisons both within the same video and across different videos. However, the pooled evaluation protocol commonly used in current VAD benchmarks makes Micro-AUROC heavily dependent on cross-video comparisons, which do not directly test whether anomalous frames rank above normal frames within the same video. To quantify this imbalance, we decompose Micro-AUROC into within-video and cross-video comparisons. Across four widely used weakly supervised VAD benchmarks, within-video pairs account for only 0.071-0.388% of all anomalous-normal frame pairs. This imbalance can make Micro-AUROC a poor indicator of within-video ranking. Video-level classifiers that assign a single score to each video achieve 81.40-97.18% Micro-AUROC while obtaining 50% Within-AUROC. For existing weakly supervised VAD detectors, replacing all frame scores in each video with their mean sets Within-AUROC to 50%. Across 120 runs, this replacement retains a median 98.8% of the original Micro-AUROC margin above chance. Because the decomposition depends on the evaluation protocol rather than the training supervision, the same distinction applies whenever VAD frame scores are pooled across videos. High Micro-AUROC alone therefore does not establish temporal localization. Our analysis reveals a limitation of pooled VAD evaluation and motivates reporting within-video ranking alongside Micro-AUROC.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video anomaly detection, Weakly supervised learning, Evaluation metrics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uLvKOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper revisits the evaluation protocol commonly used in video anomaly detection (VAD), with a particular focus on pooled frame-level Micro-AUROC. The authors decompose Micro-AUROC into two components: Within-AUROC, which measures anomalous-normal frame ranking within the same video, and Cross-AUROC, which measures ranking across different videos. They show that, on several widely used VAD benchmarks, within-video anomalous-normal pairs constitute only a very small fraction of all pairs, causing Micro-AUROC to be dominated by cross-video comparisons.

The paper supports this observation empirically in two ways. First, video-level classifiers that assign a single anomaly score to all frames of a video can still achieve high Micro-AUROC despite having 50% Within-AUROC by construction. Second, replacing all frame-level predictions in a video with their mean often preserves most of the original Micro-AUROC margin in the weakly supervised setting.

Based on these analyses, the authors argue that high Micro-AUROC alone should not be interpreted as evidence of good temporal localization and recommend reporting within-video ranking metrics such as Within-AUROC and Macro-AUROC alongside pooled metrics.

Critical strengths

The paper studies a real and relevant issue in VAD evaluation. I agree with the central observation that pooled Micro-AUROC does not necessarily reflect how well a detector localizes anomalous regions within individual videos. This is particularly relevant because Micro-AUROC is still widely used as a primary metric in VAD.

The mathematical decomposition is simple, clear, and useful. Expressing Micro-AUROC as a weighted combination of within-video and cross-video comparisons makes the behavior of the metric easy to understand. The empirical finding that the within-video pair fraction can be extremely small on common benchmarks provides a convincing explanation for why Micro-AUROC can be insensitive to within-video ranking.

The controlled experiments are also well designed. In particular, the video-level classifier experiment provides an intuitive counterexample showing that a detector can obtain high pooled AUROC without performing temporal localization. Similarly, the video-mean replacement experiment offers useful empirical evidence that existing WSVAD models can retain much of their Micro-AUROC even after their within-video ranking information is removed.

Overall, I find the empirical phenomenon identified by the paper credible, and I do not have major concerns about the correctness of the core decomposition.

Critical weaknesses

My main concern is that I do not find the practical contribution of Within-AUROC sufficiently compelling beyond its role as an analytical tool.

The paper convincingly shows that Micro-AUROC and Within-AUROC can lead to different conclusions, but a disagreement between two metrics does not establish that the ranking induced by Within-AUROC is more meaningful or more appropriate for model selection. Micro-AUROC and Within-AUROC simply emphasize different aspects of performance. In particular, pooled metrics capture global discrimination across the complete test set, including normal-only videos, while within-video metrics condition on mixed-label videos and evaluate temporal ranking within them.

More importantly, Macro-AUROC already provides a straightforward way to evaluate within-video localization by computing AUROC independently for each mixed-label video and averaging across videos. The paper itself recognizes this existing evaluation protocol. Therefore, the practical recommendation seems close to the already available strategy of reporting a global metric such as Micro-AUROC/AP together with a per-video metric such as Macro-AUROC.

I am also not convinced that the weighting used by Within-AUROC is necessarily desirable as a benchmark metric. Within-AUROC weights each video according to the number of anomalous-normal frame pairs, ainia_i n_i, whereas Macro-AUROC gives equal weight to each mixed-label video. As a result, long videos, or anomaly categories represented by longer videos, may dominate Within-AUROC. If the goal is to evaluate how reliably a method localizes anomalies across videos, equal video weighting may be more natural. The paper does not sufficiently justify why the pair-weighted aggregation of Within-AUROC should be preferred for practical model evaluation.

Relatedly, the paper reports that detector rankings can change considerably when switching from Micro-AUROC to Within- or Macro-AUROC. However, such ranking reversals are not by themselves evidence that the original ranking is incorrect. Different metrics optimize different evaluation objectives. The authors show that the rankings differ, but do not establish that Within-AUROC is better aligned with the intended downstream use of VAD than Macro-AUROC or other existing alternatives.

Finally, the broader practical insight appears limited to a point that can already be expressed using existing complementary metrics: pooled Micro-AUROC should not be interpreted as a direct measure of temporal localization, and per-video localization metrics should be considered separately. I find the analysis technically useful, but I am less convinced that it provides a sufficiently novel evaluation principle or fundamentally changes how VAD performance should be assessed.

Questions for authors
  1. What practical information does Within-AUROC provide beyond Macro-AUROC? Both metrics evaluate anomalous-versus-normal ranking within mixed-label videos, differing mainly in their aggregation weights. Beyond enabling the exact Micro-AUROC decomposition, what model behavior can Within-AUROC reveal that Macro-AUROC cannot?

  2. Why is pair-weighting by ainia_i n_i desirable for localization evaluation? Within-AUROC gives greater influence to videos containing more anomalous-normal frame pairs. This may cause long videos, or categories represented by long videos, to dominate the final metric. If the evaluation target is localization quality across test videos, why should this weighting be preferred over equal weighting as in Macro-AUROC?

  3. How should a practitioner interpret a disagreement between Micro-AUROC and Within-AUROC? The paper shows many ranking reversals, but why should the ranking given by Within-AUROC be regarded as more meaningful? Could these reversals simply reflect the fact that the metrics evaluate different properties rather than indicate a failure of Micro-AUROC?

  4. Would reporting Micro-AUROC/AP together with Macro-AUROC already address the main issue raised by the paper? Micro-AUROC/AP can characterize global discrimination including normal-only videos, while Macro-AUROC characterizes per-video localization on mixed-label videos. What additional practical decision would require Within-AUROC?

  5. Is Within-AUROC intended primarily as a diagnostic quantity or as a benchmark metric? I find it very useful for the proposed mathematical decomposition and for understanding what Micro-AUROC measures. However, I am less convinced that it is necessary as an additional standard metric. Clarifying this distinction would substantially strengthen the motivation.

Reviewer_cY3YOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper splits pooled frame-level Micro-AUROC in video anomaly detection into within-video and cross-video pairs, showing that within-video pairs make up under 0.4% of comparisons on standard WSVAD benchmarks.

Critical strengths

The finding has a direct practical consequence for how the field reports results, and the paper shows concrete ranking reversals between detectors when within-video metrics are used.

Critical weaknesses

The core insight is fairly simple and partly known: pooled AUROC being dominated by cross-group pairs is established in the fairness literature, and Macro-AUROC is already used in semi-supervised VAD for this reason, so the contribution is mostly quantifying it for WSVAD. The paper treats within-video ranking as the thing that matters but doesn’t justify why. Telling anomalous videos from normal ones is a real part of the task, Within-AUROC ignores false alarms on all-normal videos, and the interval-based results don’t clearly show Within-AUROC tracks localization better. Within-video metrics rest on a small effective number of videos (roughly 8 to 44), with no confidence intervals and known annotation disagreement, so many of the 78 reported ranking reversals could be noise between close configurations. The reproduced detectors are mostly older methods, and some reproduced scores fall well below published numbers (for example XD-Violence AP), so it’s unclear how well the conclusions carry over to current state-of-the-art models.

Questions for authors

Across the same detectors, does Within-AUROC or Macro-AUROC correlate better with event-level AP than Micro-AUROC does? This would support the claim that it’s the better measure of temporal localization.

Reviewer_zFiQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper critically examines the reliability of Micro-AUROC for evaluating temporal localization in video anomaly detection (VAD). The authors demonstrate that cross-video comparisons dominate conventional evaluation, with within-video pairs accounting for only 0.071–0.388% of all comparisons. Interestingly, video-level classifiers achieve up to 97.18% Micro-AUROC despite having no temporal localization capability. The paper advocates reporting Within-AUROC and Macro-AUROC alongside pooled metrics.

Critical strengths
  • Important research problem: The paper exposes a fundamental mismatch between the temporal localization objective of VAD and conventional evaluation protocols.

  • Strong theoretical motivation: The AUROC decomposition provides an elegant explanation of why high frame-level AUC does not necessarily indicate accurate temporal localization.

  • Convincing experiments: Video-constant predictions, score-replacement experiments, and detector-ranking reversals effectively demonstrate the limitations of Micro-AUROC.

  • Extensive evaluation: Multiple datasets, architectures, and supervision settings strengthen the generality of the findings.

Critical weaknesses
  • Limited methodological novelty: The decomposition is based on established pairwise AUROC principles, while alternative within-video metrics have previously been explored. The contribution is primarily diagnostic rather than methodological.

  • Within-AUROC does not guarantee accurate localization: It evaluates ranking but does not explicitly assess temporal boundaries, event fragmentation, or detection delays. Furthermore, its pair-weighted formulation may favor longer videos with balanced anomaly durations.

  • Cross-video discrimination remains important: The paper somewhat underemphasizes the practical importance of globally calibrated anomaly scores for consistent alert thresholds and false-alarm control.

  • Missing statistical significance analysis: Although the authors report substantial detector-ranking reversals, their statistical reliability is not established.

Questions for authors
  • How strongly does Within-AUROC correlate with event-level mAP, temporal IoU, and boundary localization accuracy? Can models with similar Within-AUROC exhibit significantly different localization performance?

  • How do Within-AUROC and Macro-AUROC change when long videos are divided into shorter clips while preserving their original frame predictions?

  • Could the authors investigate how video-specific score normalization affects Micro-AUROC versus Within-AUROC?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.