acceptodds
Under review as a conference paper at ICLR 2027

When Should Anomaly Thresholds Adapt? A Decision Framework under Prevalence Shift

Abstract

Unsupervised anomaly detectors produce continuous scores, but deployment requires thresholds that convert those scores into decisions. Reference-calibrated thresholds can become unreliable when anomaly prevalence changes, while indiscriminate batch adaptation can introduce error when the score distribution provides little evidence that adaptation is warranted. We introduce CAVET-AD (Calibrated Adaptive Value of Excess Tail for Anomaly Detection), a detector-agnostic post-hoc framework that determines whether threshold adaptation is supported by the observed score distribution. CAVET-AD calibrates scores against a frozen nominal reference, refines the upper tail using extreme-value theory, and computes an observable excess-tail response, , that controls a bounded adaptation budget. When the response is near zero, the method retains its conservative base rule rather than forcing adaptation. Across four tabular benchmarks and 320 positive-prevalence conditions, CAVET-AD increases descriptive mean F1 from to . Its behavior is strongly regime dependent: gains are substantial on CICIoT2023 and CreditCard2013, NSL-KDD shifts from fixed-favorable to adaptive-favorable as prevalence increases, and CICIDS2018 shows no benefit when Isolation Forest scores are nearly uninformative. At the natural CreditCard2013 prevalence of , adaptation is minimal and yields no statistically significant F1 improvement after multiplicity correction. A leave-one-dataset-out policy based on retains most of the adaptive benefit on a benchmark excluded from cutoff selection, while ECOD replication and detector-transfer analyses provide additional evidence of portability. Under severe controlled covariate shift, however, can increase even when adaptation is harmful. These results position CAVET-AD as a diagnostic for when prevalence-driven threshold adaptation is warranted, rather than as a universal detector of distribution shift.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
unsupervised anomaly detection, adaptive thresholding, extreme value theory, prevalence shift, tail calibration, contamination robustness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rWDUOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The author proposes CAVET-AD, a detector-agnostic post-processing threshold calibration framework: it maps detector scores to a reference tail space via a frozen reference ECDF, refines the reference upper tail using extreme value theory, and controls a bounded adaptive runtime budget with a batch tail excess response rtailr_\mathrm{tail}; when rtail approaches zero, it reverts to a conservative fixed rule.

Critical strengths
  1. The value of this work lies first and foremost in its problem framework: instead of continuing to ask "how to tune thresholds," it explicitly shifts to "when one should tune thresholds." This point holds practical significance in anomaly detection deployment.
  2. It is worth affirming that the author proactively reported multiple negative results and boundary conditions.
Critical weaknesses
  1. As currently written, this paper could easily be perceived as just "another pipeline combining EVT and adaptive budgeting," with its main selling point being the increase in average F1 score from 0.475 to 0.579.
  2. Leave-one-out dataset strategy is a cross-fitted evaluation, not a dynamic deployment on a real-time stream.
  3. The work compares SPOT, Conformal–BH, and various fixed thresholds, but does not perform a same-condition comparison with methods that directly estimate the deployment mixing ratio or contamination factor.
  4. Section 5.7 and Appendix G.5's controlled covariate shift tests show that on CreditCard2013, when shift=1.00, the average rtail rises to 0.3390, but the adaptive F1 drops sharply instead.

Some typoes:

  1. The dataset naming is inconsistent. The text alternates between CreditCard2013, CREDITCARD2013, Credit Card 2013, and CICIDS2018, CSE-CIC-IDS2018, CICIDS2018, etc. It is recommended to standardize the abbreviations and strictly adhere to them after their first appearance.
Questions for authors
  1. Can you supplement a method that directly estimates deployment prevalence or contamination proportion as a baseline, and explain under which assumptions CAVET-AD should outperform the direct estimate, and under which conditions it should underperform?
  2. The biconditional KS threshold in Appendix G.9 is a study-specific post-hoc threshold. Can it be transformed into a transferable, fixed a priori rule, or at least one validated on a second dataset? If not, should the main conclusions be further restricted to a pure prevalence-shift protocol?
Reviewer_h9EbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies threshold adaptation for unsupervised anomaly detection under prevalence shift. The proposed CAVET-AD method computes the excess tail probability of deployment scores relative to a nominal reference. It then adjusts the threshold through a bounded linear budget function and an EVT-based calibration procedure. Experiments on multiple benchmarks show F1 improvements in several prevalence regimes.

Critical strengths

The paper studies a practical problem of threshold adaptation for anomaly detection. The proposed method does not require detector retraining or labeled deployment data.

The method is simple and can be applied to different anomaly detectors.

The paper provides paired evaluations across multiple prevalence levels.

Critical weaknesses
  • The novelty seems limited. The main design computes the difference between the tail occupancy of deployment scores and a reference distribution. This difference is then mapped to a threshold budget through a linear function. Under the assumed prevalence shift, this statistic essentially serves as a scaled proxy for anomaly prevalence. The paper does not establish a new principle for deciding when threshold adaptation improves detection performance.

  • The contribution of EVT calibration is not clearly justified. For example, Table 10 shows almost identical F1 results with and without EVT refinement. This suggests that the performance gains mainly come from the linear adjustment of the operating budget. The paper also lacks direct comparisons with simpler prevalence estimation and adaptive threshold selection strategies.

  • The evidence for the proposed applicability diagnostic and detector portability remains limited. A positive excess-tail response does not always indicate beneficial adaptation. NSL-KDD shows negative effects at low prevalence, while severe covariate shift causes a substantial performance drop despite an increased tail response. The ECOD evaluation also shows improvements on only one dataset.

Questions for authors

What is the essential difference between the proposed tail-excess response and a simple prevalence-dependent threshold adjustment? Can the authors compare against direct prevalence estimation and simpler adaptive threshold rules under matched settings?

What evidence supports the claim that the excess-tail response can determine when adaptation is beneficial? How can the method explain the negative adaptation effects on NSL-KDD at low prevalence and under severe covariate shift?

How robust is the proposed method when the nominal reference contains anomalous samples? Can the authors provide evidence under natural temporal prevalence shifts rather than artificially constructed mixtures?

2 comments
Reviewer_h9EbOpenReview ↗

Response to the Rebuttal

I appreciate the authors’ time and effort for the rebuttal. Yet I’m unconvinced that the main concerns have been addressed.

For example, for Q1, the authors acknowledge that a prevalence estimate does not answer the "when to adapt" question. Besides, the results in Tables 12 and 13 seem to suggest a more nuanced relationship between tail informativeness and adaptation benefit, which I feel can raise more concerns. While the reported held-out results are encouraging, Table 14 shows that the proposed gate does not outperform adaptive. Besides, I also noticed that related concerns about EVT and prevalence baselines were also raised by other reviewers.

Overall, I appreciate the authors' clarifications, but the some concerns remain unresolved. I therefore maintain my original score.

AuthorsOpenReview ↗

Updated response to Reviewer h9Eb - Adaptation decisions and new matched comparisons

Thank you for your follow-up. We agree that our earlier response overstated what the excess-tail signal establishes about when adaptation is beneficial. A positive response alone does not establish an F1 improvement. We have narrowed the contribution accordingly and added direct comparisons to test the signal’s value.

1. What Tables 12 and 13 establish. Under pure prevalence shift, the expected tail response is approximately prevalence multiplied by the anomalous–nominal upper-tail occupancy gap. This explains why it is related to prevalence and detector informativeness, but it does not determine whether lowering a threshold improves F1. On NSL-KDD at low prevalence, additional false positives can outweigh additional detections. Under severe covariate shift, nominal score displacement can also increase tail occupancy. We therefore describe the response as controlling the size of an adjustment, whose benefit depends on the deployment regime. We no longer claim that its magnitude alone determines when adaptation helps.

Revised manuscript. Abstract, Section 1 (Contributions), Section 5.7, Appendix D.7, and Section 6.

2. Table 14 and the value of the gate. You are right that Table 14 does not establish an advantage over always adapting. Across 40 dataset–seed units, the held-out gate improved mean F1 over the fixed rule by 0.0982 (Wilcoxon p = 2.14 × 10⁻⁷). Its mean F1 was 0.0057 below always adapting; this difference was not statistically significant (p = 0.131). This supports improvement over the fixed comparator, but does not establish superiority or equivalence to always adapting.

We also added a separate held-out comparison in which tail-driven and prevalence-driven gates choose between the same fixed and full CAVET-AD thresholds. Each cutoff maximizes mean F1 on three development datasets and is evaluated on the fourth. The tail gate tied the prevalence gate on CICIDS2018 and had lower mean F1 on the other three datasets. On CICIoT2023, the respective scores were 0.6415 and 0.7037. Both prevented one harmful update, but the tail gate blocked 29 beneficial updates versus 11.

Table 14 uses the original row-index-disjoint confirmatory partitions, whereas this new comparison uses the feature-group-disjoint reconstruction. Their fitted cutoffs and results therefore differ; the CICIoT2023 tail-gate score is 0.6778 in Table 14 and 0.6415 in the new comparison. We report the comparisons separately.

Revised manuscript. Section 5.6, Table 14, and Appendix D.7, “Held-out comparison of decisions to adapt.”

3. Direct prevalence baseline and EVT. We added a median-anchored prevalence estimator on the same 320 conditions. Holding the bounded budget mapping, empirical threshold inversion, and alert rule fixed, we changed only the signal setting the budget. Each signal led on two datasets by mean F1; tail-driven minus prevalence-driven differences ranged from −0.0053 to +0.0033. These results do not support a general advantage for the tail signal. This is a simple estimator, and we did not evaluate the estimators of Scott, Jain et al., or Perini et al.

We also agree about EVT. Removing EVT refinement changes mean F1 by at most 0.00013 on any dataset, while removing adaptive budgeting costs up to 0.317. We attribute the observed gains to adaptive budgeting and no longer credit EVT with those gains.

Revised manuscript. Appendix D.7 and Table 10.

4. Additional tests of the limits. We tested nominal-reference contamination and chronological deployment. At 1% reference contamination, full CAVET-AD lost 0.0664 mean F1 on CICIoT2023, with losses in all ten seeds; the no-EVT variant showed similar sensitivity. On the chronological CreditCard2013 split, evaluated over 12 later two-hour windows, mean pooled F1 fell from 0.1151 for the fixed threshold to 0.0978 for full CAVET-AD across ten detector seeds. We report these as limitations and do not claim contamination robustness or consistent temporal-deployment gains.

Revised manuscript. Appendix D.7 and Appendix H.2.

The contribution we ask you to assess is the paired characterization of adaptation gains and failure regimes, together with the new comparisons testing whether the tail signal adds value. These comparisons also identify where it does not. In light of the additional evidence and the narrower claims, we respectfully ask you to reconsider your assessment and raise your rating to accept (4).

Reviewer_aUSEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes CAVET-AD, a post-hoc, detector-agnostic method for deciding when an anomaly threshold should adapt under prevalence shift. It uses an EVT-refined, label-free excess-tail statistic to set a bounded alert budget, and reports strong results on the four tabular benchmarks it is evaluated on.

Critical strengths
  1. The problem is well defined and important. Deciding when a deployed anomaly threshold should adapt, without labels or retraining, is a real practical issue.
  2. The method is detector agnostic and post hoc, so it can be put on top of any detector's scores cheaply.
Critical weaknesses
  1. The evaluation is too thin : Only four tabular datasets are used, and standard anomaly detection benchmarks such as ODDS (Rayana, 2016) are not, so it is hard to tell whether the results generalize.

  2. Closely related work is missing: both from the related work and as baselines, in particular adaptive thresholding with confidence sequences (Sun et al., ICML 2024) and SEAD (Shah et al., ICML 2025).

  3. The EVT ablation suggests that this component has little effect. In Table 10, removing the EVT refinement gives practically the same F1 (e.g., NSL-KDD 0.8098 vs. 0.8097), so it is unclear why it is needed.

Questions for authors
  1. Evaluation on the full ODDS benchmark: Could you report results on the full ODDS suite, with additional detectors, to show that the excess-tail diagnostic generalizes beyond the four mostly intrusion- and fraud-detection datasets used?

  2. Missing baselines (performance and positioning): How does CAVET-AD compare, both in performance under the same prevalence-shift protocol and in conceptual positioning, with existing unsupervised adaptive-thresholding methods such as Online Adaptive Anomaly Thresholding with Confidence Sequences (Sun et al., ICML 2024) and SEAD (Shah et al., ICML 2025)?

3.When to deploy: The gains seem to depend on the detector's tail being informative (Table 13) and on prevalence not being low (NSL-KDD gets worse at π ≤ 0.10, Table 12). How would a practitioner check these conditions before deployment, given that the pre-check in App. G.9 may need labeled data, and labeled anomalies are often unavailable in real deployments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.