- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper revisits the evaluation protocol commonly used in video anomaly detection (VAD), with a particular focus on pooled frame-level Micro-AUROC. The authors decompose Micro-AUROC into two components: Within-AUROC, which measures anomalous-normal frame ranking within the same video, and Cross-AUROC, which measures ranking across different videos. They show that, on several widely used VAD benchmarks, within-video anomalous-normal pairs constitute only a very small fraction of all pairs, causing Micro-AUROC to be dominated by cross-video comparisons.
The paper supports this observation empirically in two ways. First, video-level classifiers that assign a single anomaly score to all frames of a video can still achieve high Micro-AUROC despite having 50% Within-AUROC by construction. Second, replacing all frame-level predictions in a video with their mean often preserves most of the original Micro-AUROC margin in the weakly supervised setting.
Based on these analyses, the authors argue that high Micro-AUROC alone should not be interpreted as evidence of good temporal localization and recommend reporting within-video ranking metrics such as Within-AUROC and Macro-AUROC alongside pooled metrics.
Critical strengths
The paper studies a real and relevant issue in VAD evaluation. I agree with the central observation that pooled Micro-AUROC does not necessarily reflect how well a detector localizes anomalous regions within individual videos. This is particularly relevant because Micro-AUROC is still widely used as a primary metric in VAD.
The mathematical decomposition is simple, clear, and useful. Expressing Micro-AUROC as a weighted combination of within-video and cross-video comparisons makes the behavior of the metric easy to understand. The empirical finding that the within-video pair fraction can be extremely small on common benchmarks provides a convincing explanation for why Micro-AUROC can be insensitive to within-video ranking.
The controlled experiments are also well designed. In particular, the video-level classifier experiment provides an intuitive counterexample showing that a detector can obtain high pooled AUROC without performing temporal localization. Similarly, the video-mean replacement experiment offers useful empirical evidence that existing WSVAD models can retain much of their Micro-AUROC even after their within-video ranking information is removed.
Overall, I find the empirical phenomenon identified by the paper credible, and I do not have major concerns about the correctness of the core decomposition.
Critical weaknesses
My main concern is that I do not find the practical contribution of Within-AUROC sufficiently compelling beyond its role as an analytical tool.
The paper convincingly shows that Micro-AUROC and Within-AUROC can lead to different conclusions, but a disagreement between two metrics does not establish that the ranking induced by Within-AUROC is more meaningful or more appropriate for model selection. Micro-AUROC and Within-AUROC simply emphasize different aspects of performance. In particular, pooled metrics capture global discrimination across the complete test set, including normal-only videos, while within-video metrics condition on mixed-label videos and evaluate temporal ranking within them.
More importantly, Macro-AUROC already provides a straightforward way to evaluate within-video localization by computing AUROC independently for each mixed-label video and averaging across videos. The paper itself recognizes this existing evaluation protocol. Therefore, the practical recommendation seems close to the already available strategy of reporting a global metric such as Micro-AUROC/AP together with a per-video metric such as Macro-AUROC.
I am also not convinced that the weighting used by Within-AUROC is necessarily desirable as a benchmark metric. Within-AUROC weights each video according to the number of anomalous-normal frame pairs, , whereas Macro-AUROC gives equal weight to each mixed-label video. As a result, long videos, or anomaly categories represented by longer videos, may dominate Within-AUROC. If the goal is to evaluate how reliably a method localizes anomalies across videos, equal video weighting may be more natural. The paper does not sufficiently justify why the pair-weighted aggregation of Within-AUROC should be preferred for practical model evaluation.
Relatedly, the paper reports that detector rankings can change considerably when switching from Micro-AUROC to Within- or Macro-AUROC. However, such ranking reversals are not by themselves evidence that the original ranking is incorrect. Different metrics optimize different evaluation objectives. The authors show that the rankings differ, but do not establish that Within-AUROC is better aligned with the intended downstream use of VAD than Macro-AUROC or other existing alternatives.
Finally, the broader practical insight appears limited to a point that can already be expressed using existing complementary metrics: pooled Micro-AUROC should not be interpreted as a direct measure of temporal localization, and per-video localization metrics should be considered separately. I find the analysis technically useful, but I am less convinced that it provides a sufficiently novel evaluation principle or fundamentally changes how VAD performance should be assessed.
Questions for authors
-
What practical information does Within-AUROC provide beyond Macro-AUROC? Both metrics evaluate anomalous-versus-normal ranking within mixed-label videos, differing mainly in their aggregation weights. Beyond enabling the exact Micro-AUROC decomposition, what model behavior can Within-AUROC reveal that Macro-AUROC cannot?
-
Why is pair-weighting by desirable for localization evaluation? Within-AUROC gives greater influence to videos containing more anomalous-normal frame pairs. This may cause long videos, or categories represented by long videos, to dominate the final metric. If the evaluation target is localization quality across test videos, why should this weighting be preferred over equal weighting as in Macro-AUROC?
-
How should a practitioner interpret a disagreement between Micro-AUROC and Within-AUROC? The paper shows many ranking reversals, but why should the ranking given by Within-AUROC be regarded as more meaningful? Could these reversals simply reflect the fact that the metrics evaluate different properties rather than indicate a failure of Micro-AUROC?
-
Would reporting Micro-AUROC/AP together with Macro-AUROC already address the main issue raised by the paper? Micro-AUROC/AP can characterize global discrimination including normal-only videos, while Macro-AUROC characterizes per-video localization on mixed-label videos. What additional practical decision would require Within-AUROC?
-
Is Within-AUROC intended primarily as a diagnostic quantity or as a benchmark metric? I find it very useful for the proposed mathematical decomposition and for understanding what Micro-AUROC measures. However, I am less convinced that it is necessary as an additional standard metric. Clarifying this distinction would substantially strengthen the motivation.