acceptodds
Under review as a conference paper at ICLR 2027

How Much of Anomaly Segmentation Is Background? A Published 97% Pixel AUROC Can Certify Chance Inside the Object

Abstract

A published 97% pixel AUROC on liver CT certifies chance inside the organ. Anomaly segmentation, medical and industrial alike, is scored almost universally by that number over the whole image, and we show what it measures. On liver CT a Gaussian blur of the input, with no model, no training and no support set, scores 94.8% and a binary foreground mask 96.9%, within three points of published detectors; inside the organ they score 14.0% and chance. The reason is an identity: whole-image AUROC equals w*A_B + (1-w)*A_F, weighted by the fraction w of normal pixels that are background, so a reported score certifies only A_F >= (A-w)/(1-w); on liver CT w=0.94, and the claim is about w: on breast ultrasound (w=0.15) the same 97% certifies A_F >= 96.5%. Class imbalance is not the mechanism, and balancing the classes leaves the number unchanged; the partition is a reporting choice that leaves A untouched, and across 6 partition rules the liver-CT guarantee moves by 2.6 points. The clearest evidence is a single detector: masking the background out of a nearest-neighbour memory takes its whole-image score on liver CT from 97.1 to 4.5 while its foreground score barely moves, 64.7 to 63.4 — same weights, same images, only the negative class changed. Measuring every term on 53,260 images across 8 dataset families spanning medicine, industry, remote sensing and driving scenes, the inflation follows w wherever a predictor can separate the background and vanishes where it cannot. Re-running training-free detectors on the six medical benchmarks under both metrics, the ranking they induce changes wherever the frame is empty (liver CT: Kendall tau=0.12, 40 of 91 pairs flip) and holds where it is full (breast ultrasound: tau=0.98). We recommend reporting A_F beside A with the partition stated, no new metric but the same statistic on a stated set of normals, release the audit as a one-file tool, and report a training-free reference baseline both ways.

Primary area:
datasets and benchmarks
Keywords:
evaluation, metrics, pixel AUROC, AUROC decomposition, benchmark auditing, evaluation pitfalls, anomaly segmentation, anomaly detection, few-shot anomaly detection, model-free baseline, medical imaging, industrial inspection, change detection

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JyXaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the limitations of pixel-level AUROC for anomaly segmentation. The authors argue that background pixels can dominate the evaluation and lead to misleadingly high AUROC scores, even when the model performs poorly inside the foreground region. They provide a mathematical decomposition of AUROC into background and foreground terms, and evaluate this effect on multiple medical and industrial datasets. The paper also suggests reporting foreground AUROC together with the original metric.

Critical strengths

The paper discusses an important evaluation issue in anomaly segmentation. High pixel AUROC does not always mean accurate anomaly localization, and this problem deserves attention.

Critical weaknesses

Limited novelty compared with existing studies. The main motivation of this paper has already been discussed in previous works. For example, Towards Accurate Unified Anomaly Segmentation (Ma et al., WACV 2025, arXiv:2501.12295) already shows that high pixel AUROC can hide poor anomaly segmentation and argues for using pAP and DSC instead. Other works such as AUPIMO have also investigated the limitations of conventional anomaly detection metrics. Although these papers are cited, the current paper does not clearly explain its substantial difference from them. The AUROC decomposition provides a more explicit mathematical explanation, but I think the improvement in understanding is relatively limited.

The paper lacks a sufficiently new scientific finding. The main theoretical result is a straightforward decomposition of AUROC based on partitioning normal pixels into two groups. The proof is simple, and many subsequent conclusions directly follow from this identity. For example, the observation that easily separated background pixels can increase whole-image AUROC is expected. The experiments confirm this effect on more datasets, but I do not see a fundamentally new finding beyond the known limitations of AUROC. The authors also argue that class imbalance is not the actual mechanism. However, AUROC being invariant to class prevalence is already a standard property of this metric. I am not convinced that this distinction alone is a sufficient scientific contribution for ICLR.

The writing and organization need significant improvement. The abstract includes too many numerical results, dataset-specific observations, and mathematical expressions. The introduction has a similar problem. Instead of clearly explaining the research gap and motivation, it presents many detailed results before the problem is properly introduced. This makes the paper difficult to follow. In my opinion, the abstract and introduction should focus on the research question, the key idea, and the main contribution, while most equations and experimental details should be moved to later sections. The paper also mixes the metric analysis with a training-free anomaly detection baseline, which makes the main contribution less focused.

The practical contribution of the proposed evaluation protocol is not sufficiently justified. The authors recommend foreground AUROC, but previous studies have already suggested pAP and DSC to address poor anomaly localization. The paper does not clearly demonstrate why foreground AUROC should be preferred over these existing metrics. In fact, the authors show that pixel AP is much less affected by the background issue in their experiments. This makes the need for another evaluation protocol less convincing. Also, the foreground definition depends on heuristic partition rules, and its reliability across different imaging domains remains a concern.

Questions for authors

Since AUROC is already known to be insensitive to class prevalence, how is the argument about class imbalance different from existing knowledge of AUROC? Is the decomposition itself sufficient to support the claimed novelty?

Reviewer_yuGDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors identify a limitation of pixel AUROC as an evaluation metric in anomaly segmentation. They prove that pixel AUROC is a weighted average of the background AUROC (AUROC of anomalous against normal, background pixels) and the foreground AUROC (AUROC of anomalous against normal, foreground pixels, with weights determined by the fraction of normal pixels that are background/foreground pixels. In cases where the background occupies a large fraction of the image (as in liver CTs), the pixel AUROC becomes largely inflated when a method is able to distinguish anomalous pixels from normal background pixels well (scores high in background AUROC), even though it may not distinguish anomalous pixels from normal foreground pixels well (scores low in the foreground AUROC).

They demonstrate this “background effect” across many datasets and domains, and show that reporting the foreground AUROC instead of pixel AUROC changes the ranking of anomaly detection methods. They propose a training-free baseline, which serves as a reference under a specified resource budget. They also recommend papers to report foreground AUROC alongside pixel AUROC and how the partition between background and foreground is defined.

Critical strengths

● The authors show that the fraction of normal pixels that are background pixels (w) determines how strongly a reported pixel AUROC (A) constrains the minimum foreground AUROC (A_F). By computing w for a public dataset and using the pixel AUROC (A) reported by existing anomaly detection methods, they could compute the guaranteed lower bound A_F these methods achieve without having to rerun them.

● The authors show that the inflated pixel AUROC cannot be explained by class imbalance between normal and anomalous pixels alone. They randomly subsample the number of normal pixels to be equal to the number of anomalous pixels to make both classes balanced, while keeping w constant, and show that the pixel AUROC barely changes.

Critical weaknesses

● Writing: There are very serious issues about the writing. The paper frequently uses dense syntax and hard-to-parse phrases and sentences (e.g., “can certify chance”, “problem fingerprint“, “A published leaderboard, re-read.”, “the composition of the negative class is a fingerprint property pixel AUROC does not see”). This severely impacts the overall flow of the paper and the understandability of the arguments presented.

● Limited technical novelty: Although the paper provides an interesting audit of how well existing anomaly detection methods actually perform in terms of foreground AUROC across multiple datasets and domains, the main technical contributions are incremental. The decomposition of pixel AUROC into foreground AUROC and background AUROC follows from the definition of AUROC, and the lower-bound guarantee follows from this decomposition.

Questions for authors

The paper needs to be largely rewritten and I do not think this can be fixed in a revision or rebuttal.

Reviewer_17e3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether high whole-image pixel AUROC reflects accurate anomaly localization or mainly separation from easy background pixels. It derives an exact background–foreground decomposition of AUROC, explaining how background composition can conceal weak foreground discrimination. The authors validate this through model-free controls, an audit of 53,260 images across eight dataset families, and detector reranking on six medical benchmarks. On liver CT, for example, a Gaussian blur achieves 94.8% whole-image AUROC but only 14.0% foreground AUROC. The paper recommends reporting foreground and whole-image AUROC together with the stated partition, supported by an audit tool and a training-free reference baseline.

Critical strengths
  1. The decomposition distinguishes background effects from class imbalance, yielding an interpretable floor, sensitivity factor, and foreground-performance bound that motivate targeted reporting changes.
  2. The audit spans medical imaging, industrial inspection, change detection, and driving scenes. Cases with minimal inflation clarify when the effect occurs, avoiding claims that whole-image AUROC is universally misleading.
Critical weaknesses
  1. BMAD and AUPIMO already highlight AUROC’s limitations, while Dice/F1 and average precision expose the paper’s central failure case. The decomposition provides a useful explanation and retrospective bound but follows directly from AUROC’s definition. The contribution is primarily empirical, and its additional value for model selection or practical segmentation beyond existing measures remains insufficiently demonstrated.
  2. Despite convincing liver-CT validation, the retinal blur’s A_F ranges from 24.8% to 85.7% across partitions. Broader partition validation and detector-ranking stability checks are needed.
  3. Most reranking configurations are patch-memory variants. MVFA remains best under both metrics, and industrial rankings largely agree, limiting evidence for broader changes in model selection.
Questions for authors
  1. What practical insight does foreground AUROC provide beyond Dice/F1, average precision, and the limitations discussed in BMAD and AUPIMO? Can you point to a reported example where it changes a model-selection or localization conclusion that these existing measures would miss?
  2. Could you test whether detector rankings remain stable across reasonable, anatomically or semantically validated partitions beyond liver CT?
  3. Which reported ranking changes occur between distinct detector families rather than variants of patch-memory methods? Given that MVFA remains best under both metrics and industrial rankings largely agree, what evidence supports broader consequences for model selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.