acceptodds
Under review as a conference paper at ICLR 2027

When High ROC-AUC Fails: Score Collapse and the Gradient Exposure Deficit in Anomaly Detection

Abstract

In industrial anomaly detection, a detector can rank short anomalous sensor segments accurately yet produce unreliable alarms for complete operating runs when conditions change. We study this on a differential-pressure pipeline leak-detection dataset and three public time-series benchmarks. The failure mechanism is score collapse: under binary cross-entropy (BCE) training with class imbalance, normal-class scores concentrate near zero, making the alarm cutoff unreliable while the ROC-AUC (area under the receiver operating characteristic curve) remains unchanged. Score collapse produces two distinct threshold failures—cutoffs that saturate to produce false alarms on every run, and cutoffs that miss nearly all leak-containing runs—both of which pass a standard high-AUC evaluation checkpoint. We propose score-histogram entropy (, nats; entropy computed from natural logarithms) as a complement to ROC-AUC, and evaluate four failure modes requiring distinct interventions: score dispersion, probability calibration, sensitivity to head-loss modifications, and alarm-cutoff performance under operating-condition shift. Joint entropy regularisation restores score dispersion ( rises from to  nats on the pipeline dataset); post-hoc Platt scaling reduces expected calibration error from to –; and density-ratio weighted conformal prediction improves run-level recall from to at zero false alarms in a stress test under a shifted operating condition. Score collapse is reproduced across three public benchmarks and three encoder architectures; gradient exposure—not backward-call placement per se—is the operative factor in collapse prevention. Score dispersion should be evaluated alongside ROC-AUC before a detector is used in a thresholded deployment setting.

Primary area:
interpretability and explainable AI
Keywords:
anomaly detection, score collapse, class imbalance, conformal prediction, multi-task learning, optimization dynamics, model calibration, distribution shift, evaluation metrics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_D7suOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper criticizes the ROC-AUC score as a metric for Time Series Anomaly Detection, and suggests a secondary metric.

Critical strengths

None

Critical weaknesses

ROC-AUC scores are not used in modern time series anomaly detection, as they ignore correlations between anomalies at similar timesteps. E.g. see https://www.sciencedirect.com/science/article/pii/S0925231226015523 or https://dl.acm.org/doi/10.14778/3551793.3551830 .

So finding that ROC-AUC are not suitable to time series anomaly detection is not a scientific contribution.

Questions for authors

None. I see no way to convince me to accept this paper. If I missed some contribution besides the known criticism of ROC-AUC scores and the suggestion of a metric that is not compared to research standards, please point this out to me, and also please make this contribution clear for the next version of this paper.

Reviewer_6MqmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why an anomaly detection algorithm trained with class-imbalanced data with binary cross-entropy can have a good rank (ROC-AUC) but have unreliable final alarms. It points the root cause to a score collpase phenomenon, with normal window scores nearly 0. Then the explored several strategies to resolve this problem. Experiments on a proprietary pipeline and three public datasets supports the existence of such a problem.

Critical strengths
  1. The paper is candid about its scope and has a very targeted discussion on the problem.

  2. The code and results are released for public benchmark experiments.

  3. The discussed issue can serve an educational purpose for the community.

Critical weaknesses
  1. The paper reads like a technical report rather than a developped contribution. The main evidence is from a proprietary pipe rig with only 31 runs and a 33-parameter linear head on a frozen encoder. The phenomenon needs much more extensive experiments to support its prevalence. Those public setups are too small to support the claim.
  2. The mechanism claim, that the amount of gradient the constraint receives, is not shown. For example, it seems to show that, say in Table 15, just adding the constraint, even without the update, prevents the collapse already.
  3. Some experimental results contradict each other. The same "separate" model is listed as collapsed in Table 3 (entropy 0) but as not collapsed in Table 7 (entropy ~1.50).

Minor:

The supported theory is a bit far-fetched and unrelated to the key claim. It does not explain why trained networks collapse while preserving ranks.

The paper has disproportionate figures and scattered tables. The presentation is relatively poor.

Questions for authors

See the weaknesses above.

Reviewer_xsRSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Industrial anomaly detectors commonly produce thresholded alarms for complete operating runs. High window-level ROC-AUC can conceal unreliable alarm behavior under changing operating conditions. This paper proposes score-histogram entropy as a diagnostic and evaluates entropy regularization, probability calibration, and density-ratio-weighted conformal thresholding. Experiments on a pipeline dataset and three public benchmarks report increased score dispersion and shifted-condition recall improvements from 1/11 to 9/11 runs.

Critical strengths
  1. Evaluating run-level false alarms and detection alongside window-level ROC-AUC addresses an operationally important mismatch.
  2. Distinguishing score dispersion, probability calibration, and distribution-shift robustness provides a useful framework for organizing detector diagnostics.
  3. The experiments include multiple public datasets, architectures, seeds, and optimization controls, while explicitly acknowledging the limitations of shifted-condition coverage claims.
Critical weaknesses
  1. The central causal explanation conflicts with quantile invariance: strictly increasing score compression can make histogram entropy zero while preserving ROC-AUC and every recalibrated run-level alarm.
  2. Table 6 reports 11/11 recall despite a minimum post-onset run maximum of .001 falling below the .003 cutoff, contradicting the stated detection rule.
  3. The “matched-exposure” C3 control performs 150 constraint updates per BCE update, changing regularization exposure and AdamW dynamics rather than matching joint training.
  4. Standard calibration uses run maxima, whereas Equation 2 and the reported effective sample size use window-level observations, confounding density weighting with a change in calibration unit.
  5. Appendix N reports zero incremental diagnostic value over simpler dispersion statistics, while the partial-collapse argument relies on an arbitrary standard-deviation threshold rather than a matched comparison.
Questions for authors
  1. What mechanism beyond score compression explains threshold failure, given that strictly monotone transformations preserve quantile-based alarm decisions?
  2. Can you reconcile Table 6’s run maxima with its recall counts and explain why the “Separate” condition has zero entropy in Table 3 but nonzero entropy in Table 7?
  3. Does separate optimization match joint training when using one constraint update per BCE update under a controlled optimization budget?
  4. Does density weighting retain its benefit when weighted and unweighted methods use identical run-level calibration units?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.