- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the limitations of pixel-level AUROC for anomaly segmentation. The authors argue that background pixels can dominate the evaluation and lead to misleadingly high AUROC scores, even when the model performs poorly inside the foreground region. They provide a mathematical decomposition of AUROC into background and foreground terms, and evaluate this effect on multiple medical and industrial datasets. The paper also suggests reporting foreground AUROC together with the original metric.
Critical strengths
The paper discusses an important evaluation issue in anomaly segmentation. High pixel AUROC does not always mean accurate anomaly localization, and this problem deserves attention.
Critical weaknesses
Limited novelty compared with existing studies. The main motivation of this paper has already been discussed in previous works. For example, Towards Accurate Unified Anomaly Segmentation (Ma et al., WACV 2025, arXiv:2501.12295) already shows that high pixel AUROC can hide poor anomaly segmentation and argues for using pAP and DSC instead. Other works such as AUPIMO have also investigated the limitations of conventional anomaly detection metrics. Although these papers are cited, the current paper does not clearly explain its substantial difference from them. The AUROC decomposition provides a more explicit mathematical explanation, but I think the improvement in understanding is relatively limited.
The paper lacks a sufficiently new scientific finding. The main theoretical result is a straightforward decomposition of AUROC based on partitioning normal pixels into two groups. The proof is simple, and many subsequent conclusions directly follow from this identity. For example, the observation that easily separated background pixels can increase whole-image AUROC is expected. The experiments confirm this effect on more datasets, but I do not see a fundamentally new finding beyond the known limitations of AUROC. The authors also argue that class imbalance is not the actual mechanism. However, AUROC being invariant to class prevalence is already a standard property of this metric. I am not convinced that this distinction alone is a sufficient scientific contribution for ICLR.
The writing and organization need significant improvement. The abstract includes too many numerical results, dataset-specific observations, and mathematical expressions. The introduction has a similar problem. Instead of clearly explaining the research gap and motivation, it presents many detailed results before the problem is properly introduced. This makes the paper difficult to follow. In my opinion, the abstract and introduction should focus on the research question, the key idea, and the main contribution, while most equations and experimental details should be moved to later sections. The paper also mixes the metric analysis with a training-free anomaly detection baseline, which makes the main contribution less focused.
The practical contribution of the proposed evaluation protocol is not sufficiently justified. The authors recommend foreground AUROC, but previous studies have already suggested pAP and DSC to address poor anomaly localization. The paper does not clearly demonstrate why foreground AUROC should be preferred over these existing metrics. In fact, the authors show that pixel AP is much less affected by the background issue in their experiments. This makes the need for another evaluation protocol less convincing. Also, the foreground definition depends on heuristic partition rules, and its reliability across different imaging domains remains a concern.
Questions for authors
Since AUROC is already known to be insensitive to class prevalence, how is the argument about class imbalance different from existing knowledge of AUROC? Is the decomposition itself sufficient to support the claimed novelty?