SubSelect: Unsupervised Model Selection for Anomaly Detection Using Subset Contrast

Unsupervised anomaly detection involves selecting an algorithm and its hyperparameters, but the absence of labels makes it difficult to validate this choice. We propose SubSelect, an unsupervised model selection approach for anomaly detection that uses only the models' anomaly scores and unlabeled data. It compares the highest-scoring observations, meaning the most anomalous ones according to the respective model, with the remaining data. If a model separates anomalies and normal observations well, its highest-scoring observations should differ from the remaining data in feature space. We quantify the contrast between the highest-scoring and the remaining observations with a combination of complementary measures. To avoid choosing a single top subset fraction, we aggregate the results across multiple fractions. We evaluate our approach against 34 unsupervised model selection methods and two baselines on 39 datasets, using 297 candidate anomaly detection models. On average precision (AP), SubSelect attains the best mean rank and outperforms 34 of the 36 compared approaches in pairwise Bayesian signed-rank tests, with no comparison decided against it. Our selections are not significantly worse than always choosing the 54th-best model out of the 297 models, compared with the 80th-best for the next selection method. SubSelect is the only method in our comparison that significantly exceeds the expected AP of an Isolation Forest baseline with randomly chosen hyperparameters. For ROC-AUC, SubSelect also significantly outperforms the Isolation Forest baseline, attains the best mean rank, and outperforms the majority of other methods in pairwise Bayesian signed-rank tests, with no comparison decided against it.