acceptodds
Under review as a conference paper at ICLR 2027

SubSelect: Unsupervised Model Selection for Anomaly Detection Using Subset Contrast

Abstract

Unsupervised anomaly detection involves selecting an algorithm and its hyperparameters, but the absence of labels makes it difficult to validate this choice. We propose SubSelect, an unsupervised model selection approach for anomaly detection that uses only the models' anomaly scores and unlabeled data. It compares the highest-scoring observations, meaning the most anomalous ones according to the respective model, with the remaining data. If a model separates anomalies and normal observations well, its highest-scoring observations should differ from the remaining data in feature space. We quantify the contrast between the highest-scoring and the remaining observations with a combination of complementary measures. To avoid choosing a single top subset fraction, we aggregate the results across multiple fractions. We evaluate our approach against 34 unsupervised model selection methods and two baselines on 39 datasets, using 297 candidate anomaly detection models. On average precision (AP), SubSelect attains the best mean rank and outperforms 34 of the 36 compared approaches in pairwise Bayesian signed-rank tests, with no comparison decided against it. Our selections are not significantly worse than always choosing the 54th-best model out of the 297 models, compared with the 80th-best for the next selection method. SubSelect is the only method in our comparison that significantly exceeds the expected AP of an Isolation Forest baseline with randomly chosen hyperparameters. For ROC-AUC, SubSelect also significantly outperforms the Isolation Forest baseline, attains the best mean rank, and outperforms the majority of other methods in pairwise Bayesian signed-rank tests, with no comparison decided against it.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
anomaly detection, outlier detection, unsupervised learning, model selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gbVxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper selects an anomaly detector without labels. For each candidate model, it takes the top-scored fraction of points and measures how far they lie from the rest of the data, using Mahalanobis distance and a Gaussian KDE fitted on the rest. The candidates are ranked on each measure and the ranks are averaged over 50 top fractions (1% to 50%). On the benchmark of Ma et al. (2023) with 39 datasets and 297 models, it has the best mean AP rank among 37 approaches. It is the only selection method that beats the Isolation Forest baseline (IFOREST-R) significantly on AP, but the gain is small.

Critical strengths
  1. Simple rule that needs no labels, no reference datasets and no per-dataset tuning.
  2. Careful statistics (Holm correction, Bayesian signed-rank test, Friedman-Nemenyi) on a public benchmark.
  3. Thorough ablations (measures, fitting set, fraction grid, pool size) and honest reporting of weak spots.
Critical weaknesses
  1. The novelty is modest. Rating a model by how separable its top-ranked points are in feature space is the core idea of IREOS and SIREOS. SubSelect replaces the classifier with MD and KDE and adds rank aggregation over fractions. This is a sensible step, but not a new principle.

  2. The contrast measures are themselves standard anomaly detectors, and SubSelect rewards candidates whose top points look anomalous under MD and KDE. It may simply favour candidates that agree with an MD or KDE detector. Please report the AP of plain MD and KDE detectors on all 39 datasets, and the AP of selecting the candidate whose ranking best matches them.

  3. The algorithms that get selected are never shown. In high dimension the Scott bandwidth h = |R|^(-1/(d+4)) is close to 1 (0.996 for InternetAds, d = 1,555), so the KDE term should behave like a nearest-neighbour distance. Table 9 fits this: removing kNN hurts most, and removing Isolation Forest raises AP. A table of how often each of the eight algorithms is picked, against the best model per dataset, would settle it.

  4. The margins are small. The gain over IFOREST-R is 0.025 AP, and over MC-NDCG 0.020. Against MC-NDCG it wins 18 datasets, ties 2 and loses 19 (Table 10). HITS-auth has the higher mean ROC-AUC (0.785 vs 0.779). The paper should say plainly that it does not beat these two decisively.

  5. The datasets are not independent, yet all tests treat the 39 as independent. Pima (ODDS) and Pima (DAMI) have identical n, d and anomaly counts (768, 8, 268), and Annthyroid, Glass, Arrhythmia and the breast-cancer sets also appear twice in some form (Table 3).

  6. Assumption A1 (anomalies are rare) fails on a quarter of the data. Ten of 39 datasets have more than 20% anomalies, e.g. Arrhythmia (DAMI) at 45.8%. There, the top subset at small p is mixed and the reference holds many anomalies. Results should be broken down by anomaly rate.

  7. Everything is evaluated on the 297 classical models of one benchmark. Section 6 says the method applies to deep detectors, but this is not tested. Related work on hyperparameter optimization and LLM-based model selection for outlier detection is missing [1], [2], [3], and the review of outlier model selection [4] should be discussed.

  8. The total cost is not reported. From Appendix A.4.5 I estimate about 121 single-core hours over the 39 datasets (67 for KDE on ALOI). Please give the total over all 297 models and compare it with the consensus methods.

References

[1] Zhao, Y. and Akoglu, L. (2020). Hyperparameter Optimization for Unsupervised Outlier Detection (HPOD). arXiv preprint.

[2] Chen, S. et al. (2024). PyOD 2: A Python Library for Outlier Detection with LLM-powered Model Selection. arXiv preprint.

[3] Yang, T. et al. (2024). AD-LLM: Benchmarking Large Language Models for Anomaly Detection. arXiv preprint.

[4] Ma, Q. et al. (2023). The Need for Unsupervised Outlier Model Selection: A Review and Evaluation of Internal Evaluation Strategies. SIGKDD Explorations, 2023.

Questions for authors
  1. How often does SubSelect pick each of the eight algorithms, and how does that compare with the best model per dataset?

  2. What AP do plain MD and KDE detectors reach, and what if you select the candidate that best matches them?

  3. Do the AP results against IFOREST-R and MC-NDCG survive after removing duplicated sources?

  4. How does it perform on the 10 datasets with more than 20% anomalies compared with the other 29?

  5. Did you try other bandwidth rules, or a kNN distance as a third measure?

4 comments

W1. IREOS and SIREOS

We agree that IREOS and SIREOS are the closest methods. Our contribution is not the general use of input features, but the ranking-based subset comparison. IREOS uses classifier-based separability for individual observations, while SIREOS uses pairwise similarity. SubSelect forms top and reference subsets from each model's ranking, fits measures on the reference and evaluates the top subset. It combines measures and aggregates across top subset fractions. No compared method combines all three components (Section 2). Ablations support this design (Section 5.3).

W2/Q2. MD/KDE controls

To test these simpler alternatives, we added Table 16 (Section A.4.5). MD and KDE fitted and scored on all data give mean AP of 0.3250 and 0.3213. Selecting the candidate whose observation ranking best matches theirs by Spearman correlation gives 0.3509 and 0.3517, against SubSelect's 0.4241. All four also have lower mean ROC-AUC, supporting subset-based selection over these simpler uses of MD and KDE.

W3/Q1. Algorithm selections

The new Table 7 gives SubSelect's selection counts for kNN (14), Isolation Forest (10), LOF (9), OCSVM (3), LODA (2), HBOS (1), ABOD (0) and COF (0). It also gives MD-only/KDE-only counts and algorithms with the best AP/ROC-AUC models.

KDE can act like nearest-neighbor distance when the nearest reference point dominates its estimate. However, MD-only selects kNN more often than KDE-only (14 versus eight datasets), so this preference is not specific to KDE.

W4. Close comparisons

Section 5.2 now reports the mean AP gain over MC-NDCG of 0.020, about 5%, alongside 18 higher, 2 equal, and 19 lower results. SubSelect also has the better mean AP rank (16.01 versus 16.59, Table 1) and is the only compared method to significantly outperform iForest-r on AP. HITS-auth has higher mean ROC-AUC (0.7852 versus 0.7786), while SubSelect has the better mean rank (13.19 versus 16.13, Table 6). Neither Bayesian comparison reaches 0.95 probability in either direction.

W5/Q3. Dataset dependence

You are right that some datasets are not independent, as acknowledged in Section A.3.3. We kept Ma et al.'s models and datasets for comparability and to test for a significant AP gain over iForest-r, which none of their compared methods achieved. Table 11 separates ODDS and DAMI, rather than removing shared sources. Mean AP gains are +0.037 and +0.012 over iForest-r, and +0.035 and +0.002 over MC-NDCG, respectively.

W6/Q4. High anomaly ratios

We agree that datasets that violate A1 are challenging for SubSelect. To mitigate this, SubSelect aggregates across larger subset fractions that accommodate high anomaly ratios. On the 29 datasets with at most 20% anomalies, new Tables 12 and 13 show larger mean AP and ROC-AUC gains over iForest-r. Above 20%, the AP gain becomes much smaller and mean ROC-AUC falls slightly below iForest-r.

W7. Deep models and related work

Section 6 now clarifies that SubSelect can accept deep-model scores, but deep-model selection remains untested. We added hyperparameter optimization and LLM-assisted selection to Section 2 and evaluated PyOD 2 [2] and AD-AGENT Selector. Prior benchmark knowledge from training could inflate LLM performance. This is one reason we initially omitted them.

AD-LLM's dataset-informed selection uses labeled examples, unavailable in our setup [3]. HPOD [1] tunes one algorithm rather than selecting among them. Section 2 discusses it and Ma et al.'s stand-alone and consensus strategies [4].

W8. Runtime

Section A.4.6 now gives total estimates. Scaling one representative candidate's timings to all 297 models and 39 datasets gives 10.5 single-core hours for MD, 110.5 for KDE and 120.9 combined, excluding candidate training and prediction. An application usually concerns one dataset, with a median combined estimate of 12.2 minutes for 297 models and 50 fractions. Models, datasets, fractions and measures can run in parallel.

For consensus methods whose results we reuse, Ma et al. report MC below one hour on ALOI and MCS, Ens and HITS within ten minutes per dataset (their Figure 1). These use model outputs without input features. IREOS, a stand-alone method using input features like SubSelect, was the slowest, taking over 16 days on ALOI. Their timings provide context but were obtained on different hardware.

Q5. Bandwidth and kNN

New Table 15 compares Scott and Silverman rules at multipliers 0.5, 1, and 2. Mean AP ranges from 0.4157 to 0.4253 and ROC-AUC from 0.7730 to 0.7786. Our unchanged Scott default gives 0.4241 AP and 0.7786 ROC-AUC.

We focused on MD and KDE to combine global distance with local density rather than adding kNN's local distance and its choice of k as another parameter. kNN and other contrast measures remain possible additions or alternatives for future work (Section 6).

Dear Reviewer,

Thank you for your feedback. We would appreciate it if you could confirm before the end of the discussion period whether our responses address your concerns.

Best regards,

The authors

Thanks to the authors for the detailed reply and for running the extra analyses. I could only check it against the submitted PDF, so the new tables are taken on trust for now.

  • Novelty: I still see SubSelect as a sensible step beyond IREOS and SIREOS rather than a new principle, but the design is clearly described and I am fine with how Section 2 positions it.

  • MD and KDE controls: This was my main worry, and the new controls answer it well. If plain MD and KDE, and selecting by agreement with them, both fall well short of SubSelect, then it does more than copy a density detector. Please keep these results in the paper.

  • Selected algorithms: The counts are useful, and kNN is picked much more often than its share of the pool would suggest. How does this compare with the algorithms behind the best models? Since MD alone shows the same preference, I agree my KDE argument does not explain it.

  • Margins and dataset overlap: The revised wording on MC-NDCG and HITS-auth is fair. The ODDS/DAMI split helps, but on DAMI the gain over MC-NDCG is close to zero. It would be good to say this next to the significance claims, and to state that overlapping datasets are counted as independent.

  • Anomaly rate: The breakdown confirms that the method is weaker when anomalies are not rare. Could this go into the limitations section?

  • LLM-based selection: Good to see PyOD 2 and AD-AGENT added. How did they do? The mean ranks quoted in the reply also differ from the submitted Table 1, so I assume the ranking now includes more methods. Is SubSelect still first?

  • Related work: I withdraw my remark on Ma et al. (2023), since the paper is built on that benchmark.

  • Runtime and bandwidth: The totals and the bandwidth check answer my questions.

Overall, the reply is responsive and the new controls make the result more convincing. The novelty is still incremental and the gains are small, so I keep my rating at weak accept.

We thank the reviewer for the re-assessment and detailed reply.

Selected algorithms. Table 7 in Section A.4.4. of the revised manuscript compares in the "Best AP" and "Best ROC-AUC" columns the number of datasets on which each algorithm was best-performing among all 297 models.

Margins and dataset overlap. We will (i) state next to the significance claims in Section 5.2 that the advantage over MC-NDCG is concentrated on the ODDS datasets (mean AP difference 0.002 on DAMI, Table 11), and (ii) state in Section 4.2 explicitly that the tests treat the 39 datasets as independent although some share a common source, which we retain for comparability with Ma et al. (2023).

Anomaly rate. We will mention the weaker results at higher anomaly proportions in the limitations section consistent with the minority assumption (A1), phrased as a descriptive finding.

LLM-based selection. PyOD 2 attains mean AP 0.372 and mean ROC-AUC 0.726, AD-AGENT Selector 0.362 and 0.731 (see Tables 1 and 6 of the revised manuscript). Both comparisons are decided in favor of SubSelect. The quoted mean ranks differ from the submitted Table 1 because the rank pool grew from 37 to 45 approaches. SubSelect is still first: it retains the best mean rank on both metrics with no comparison decided against it.

We will incorporate the proposed changes in the revised manuscript and thank the reviewer for the suggestions that have strengthened the paper.

Reviewer_ZpkdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript proposes SUBSECLECT, a model selection method for unsupervised anomaly detection. The core of this method lies in scoring based on the abnormal data distribution of each model, and evaluating the top subset fraction using multiple measures. SUBSELECT compared 34 methods and two baseline methods on 39 datasets and achieved the best performance.

Critical strengths

1.The proposed method takes the unlabeled data and the anomaly scores of each model as inputs, without any additional preprocessing cost. 2.The proposed method is based on statistical tools and has a clear logic, making it easy to reproduce.

Critical weaknesses

1.The methods from 2024 to 2026 are only LOTUS, which is not sufficient to demonstrate the advantages of SUBSELECT. Moreover, LOTUS searches for algorithms and hyperparameter configurations on historical datasets. Restricting it to a fixed pool of 297 candidate configurations may limit the benefits of this search, raising concerns about whether the comparison fully reflects the capabilities of the original method. 2.Although the experiments have shown that the proposed solution is highly effective, I have doubts about the validity of SUBSELECT, especially when the performance of the selected anomaly detection methods is poor, such as when the AUROC is lower than 0.8. In such cases, the differences in the anomaly distribution among them are significant. However, the article does not explain how this method is effective in such situations, and there is also a lack of theoretical proof in the paper.

Questions for authors

1.Could the authors include additional recent model selection methods from 2024–2026, or explain why LOTUS is the only recent method included? 2.How does restricting LOTUS to the fixed pool of 297 candidate configurations affect its performance relative to its original search procedure? Could the authors provide an experiment or analysis assessing the impact of this adaptation? 3.Could the authors analyze the relationship between SUBSELECT scores and the AP/AUROC of candidate models within each dataset, especially when detection performance is low?

2 comments

W1/Q1. Recent methods and comparison categories

We added five recent model selection approaches from 2024 to 2026. These extend the comparison with AutoUAD's RTM/EAG criteria, meta-learning ensembles through MetaEns, and LLM-assisted selection through PyOD 2 and AD-AGENT Selector. Tables 1 and 6 now compare 45 approaches, including SubSelect, both baselines, broader-search LOTUS, discussed under Q2, and two broader-search AutoUAD variants.

ApproachMean APMean ROC-AUC
AutoUAD-RTM (fixed)0.28860.6693
AutoUAD-EAG (fixed)0.34810.7022
AutoUAD-RTM (search)0.28220.5506
AutoUAD-EAG (search)0.36210.7020
MetaEns0.38380.7592
PyOD 20.37240.7259
AD-AGENT Selector0.36190.7314
SubSelect0.42410.7786

Tables 1 and 6 also show that SubSelect retains the best mean rank on both metrics. The Bayesian test decides 42 of 44 AP comparisons and 32 of 44 ROC-AUC comparisons in its favor, with none against it. The experimental settings are described in Section A.3.3.

W2. Selection on difficult datasets

We have added two analyses that separate two causes of low ROC-AUC: the absence of any good candidate model, and SubSelect selecting a weaker model when better alternatives were available (see updated Tables 12 and 13).

On the ten datasets where even the best candidate has ROC-AUC below 0.8, mean AP is 0.3723 for SubSelect against 0.3659 for iForest-r, and mean ROC-AUC is 0.6070 against 0.6000. The gains are small, but SubSelect still exceeds the strongest baseline on average on these datasets.

SubSelect's selected model has ROC-AUC below 0.8 on 17 datasets, comprising the ten datasets where no candidate reaches 0.8 and seven datasets where a model reaching at least 0.8 was available but not selected. On these 17 datasets, SubSelect's mean ROC-AUC difference from iForest-r is -0.0149. These numbers describe where SubSelect struggled but do not predict how often it will struggle on new datasets. Aggregating over top subset fractions up to p=0.50p = 0.50 avoids relying on one small top subset. The aggregation ablation (Section 5.3) shows that restricting the grid to small fractions harms mean AP rank most.

We have no theoretical guarantee of correct selection and have clarified in Section 6 where the contrast measures can fail. The correlation analysis under Q3 examines how selection scores relate to performance.

Q2. LOTUS with a broader search

LOTUS (fixed), the submitted comparison, selects from the fixed 297 configurations on the nearest reference dataset without running GAMA. We have now added LOTUS's GAMA search over 25,393 configurations within the same eight algorithm families and original hyperparameter bounds. Integer parameters vary in steps of one and the three real-valued parameters in steps of 0.01. Search maximizes reference ROC-AUC using the submitted nearest-reference mapping with a two-hour budget per reference, following the authors' released experiment code (see A.3.3).

LOTUS settingMean APMean ROC-AUC
LOTUS (fixed)0.34950.7184
LOTUS (search)0.34860.7166

Broader search improves AP on 19 datasets, ties on 5, and lowers it on 15. The ROC-AUC counts are 21, 7, and 11. Although more datasets improve than decline, the average improvements are smaller than the declines. The overall means are therefore slightly lower. Restoring LOTUS's search procedure therefore did not improve mean performance under this benchmark setup. Tables 1 and 6 now include LOTUS (search) alongside LOTUS (fixed).

Q3. Selection scores and model performance

New Table 14 (Section A.4.5) reports Spearman correlations between selection scores and performance across the 297 models within each dataset. We use negative aggregate rank as the selection score, so higher values indicate preferred models.

Dataset groupMedian AP correlationMedian ROC-AUC correlation
All 390.4460.549
Best candidate ROC-AUC below 0.80.2470.414
SubSelect-selected ROC-AUC below 0.80.1850.238

Correlations are positive on 33 of 39 datasets for AP and 35 for ROC-AUC, with positive but smaller median correlations in both low-performance groups.

Dear Reviewer,

Thank you for your feedback. We would appreciate it if you could confirm before the end of the discussion period whether our responses address your concerns.

Best regards,

The authors

Reviewer_8ze9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

It proposes an unsupervised anomaly detection model selection approach called 'SUBSELECT' using anomaly scores and unlabelled data only. It does not require any labelled/ground-truth anomalies. It is based on the assumption/fact that anomalies have large distances from normal samples and have lower densities compared to normal samples. It attempts to quantify contrast between anomalies and normal samples using Mahalanobis distance (MD) and kernel density estimation (KDE). Because we do not know what percentage of samples are anomalies, it considers different percentiles and aggregates the contrast between anomalies and normal data. It compares SUBSELECT with different existing model selection approaches using 297 candidate anomaly detection models (with different hyperparameter settings for 8 well-known traditional anomaly detection methods) on 39 datasets.

Critical strengths
  • The problem it attempts to tackle is interesting and challenging.
  • A lot of experimental results.
  • The proposed idea is simple (even though I feel like the intuition is not clear).
Critical weaknesses
  • All of the experimental methodologies, design, and results were taken from Ma et al. (2023), including the anomaly scores of all methods and datasets. The only contribution of the paper appears to be computing contrast based on MD and KDE to identify which model produces better results.
  • It is not clear how MD and KDE capture complementary information. The rationale and intuition for using them are not clearly explained. To me, they capture similar information — samples with low density generally have large distances to most samples.
  • It assumes that anomalies are significantly different from the majority of data in the feature space (i.e., their distances to most samples are high and they have low density). This assumption is often violated in practice, where normal data often have varying density clusters, and anomalies are often significantly different locally (in their neighbourhood) only.
  • MD computes the centre of RR as the mean feature values of the samples. Is it really meaningful in the case of arbitrarily shaped multimodal/multicluster normal data?
  • I am not sure about the practical implications of this method. All the evaluations in the paper are based on benchmark data where labels are available to do the evaluation at the end. It can not be guaranteed that the selected model will do better on unseen data; i.e., generalisation cannot be ensured.
Questions for authors
  • How about computing r(m,p)r(m,p) using rankings of sample in TT based on anomaly scores of model mm? This way, we do not need additional computational cost to compute MD and KDE, which can be quite expensive in the case of large nn and dd. MD and KDE also give measures of the outlierness of samples.
  • Would you please provide some scenarios where MD and KDE are good and not so good so that we understand why we need both of them?
  • How sensitive is hph_p used in KDE? I am not sure if the ablation study for this has been provided.
  • I wonder why the 'Cluster-quality indices' method mentioned in Table 5 is not used in the main experimental results discussed in Table 1?
  • Do you think this approach works reasonably well in practice, where normal data often have arbitrarily shaped, varying-density multiple clusters, and we may have local and clustered anomalies?
  • The denominator in Eqn 3 should be |M|, right? I believe MM is a set of candidate models.
3 comments

W1. Contribution and benchmark reuse

Our main contribution is how we compare the highest-scoring observations with the remaining data. For each candidate model and top subset fraction, SubSelect fits multiple contrast measures on the remaining observations and evaluates the highest-scoring observations against them. Combining measures accounts for different ways anomalies can differ from normal data, while aggregation across fractions avoids choosing one cutoff without labels. Our initial ablations (Section 5.3) support both design choices. Sections 2 and 3 explain the difference from other feature-based approaches.

We use Ma et al.'s datasets, anomaly scores, and results for 30 selection variants, but compute our selections, ablations, and additional comparisons ourselves. The initial submission added SIREOS and meta-learning with MetaOD, ELECT, and LOTUS. The revision adds AutoUAD's RTM/EAG criteria, the meta-learning ensemble selector MetaEns, and LLM-assisted selection with PyOD 2 and AD-AGENT Selector. Tables 1 and 6 cover 45 approaches, including fixed-pool and broader-search AutoUAD comparisons.

W2/Q2. Complementary measures

Figure 2 shows two synthetic examples where the measures respond differently. KDE identifies anomalies in the low-density gap between normal clusters that MD misses because they lie near the overall mean. MD captures departures from a narrow correlated normal band that KDE with one bandwidth can smooth over. Figure 7 (Appendix A.4.3) shows three benchmark datasets where MD selects a strong model and KDE a weaker one (row 1), the reverse (row 2), and both select weaker models than the combination (row 3). The combination shows more gains than losses against each measure on both metrics (Table 2, Appendix Table 8, Appendix Figures 8 and 9). We have clarified the rationale for MD and KDE in Section 3.3.

W3/Q5. Local anomalies and practical limitations

We agree that local anomalies and normal clusters with different densities can be difficult for both measures. Anomalies can lie near the reference mean and still have higher density than a sparse normal cluster. Section 6 now explains this limitation and discusses further contrast measures that could be used within the same selection procedure.

Across the 39 datasets studied, SubSelect attains the best mean rank among 45 approaches and is the only compared method that significantly outperforms iForest-r on AP. As a separate check, on the ten datasets where even the best candidate model has ROC-AUC below 0.8, SubSelect attains mean AP 0.3723 against 0.3659 for iForest-r (Tables 12 and 13, Section A.4.5).

W4. The reference mean

The mean does not necessarily represent a typical normal observation in multicluster data. MD can therefore give low contrast to anomalies between clusters, while KDE identifies their low reference density (Figure 2, Section 3.3). This motivates combining measures rather than relying on MD alone. Table 2 and Figure 7 (Appendix A.4.3) show empirical benefits of the combination.

W5. Unseen data

The task is to select a model for an existing unlabeled dataset, with models fitted and scored on those same observations (Section 4.1). Dataset labels are used to evaluate the selections.

SubSelect uses one fixed configuration across all 39 datasets, with no dataset-specific adjustment of selection hyperparameters. We have clarified in Section 6 that our evaluation does not indicate performance on future observations.

Q1. Using scores alone

SubSelect uses input features to check whether a model's highest-scoring observations differ from the remaining data. This adds information beyond their positions in the model's own ranking. Tables 1 and 5 include methods that use scores alone, so we compare against approaches that avoid this additional computation. Sections A.2 and A.4.6 report the computational cost of MD and KDE.

Q3. Bandwidth sensitivity

We added a KDE bandwidth ablation (Table 15), comparing Scott's and Silverman's rules with multipliers 0.5, 1 and 2 on all 39 datasets and 297 models. Mean AP ranges from 0.4157 to 0.4253 and mean ROC-AUC from 0.7730 to 0.7786. Average performance is therefore relatively stable across these settings, although individual datasets can be more sensitive. We retain Scott's rule as our default, with mean AP 0.4241 and mean ROC-AUC 0.7786.

Q4. Cluster-quality indices in Table 1

They are included as std, rs, h, ch, s, i, db, xb, d, and sd, with their grouping explained in Table 5 (Section A.3.3). They compare high- and low-score groups. Even though they receive the true anomaly count to define the split, none of them significantly outperforms Random on AP, although this does not rule out better score-only criteria.

Q6. Equation 3

The symbol M\mathcal{M} refers to the set of models and M=∣M∣M=|\mathcal{M}| to its size, so division by MM was intended. To simplify the notation, we now use ∣M∣|\mathcal{M}| instead of MM.

I thank the authors for their effort in responding to my concerns and questions. Their responses answer some of my concerns/questoins while others are not answered adequately. After reading fellow reviewers' comments and the authors' responses, I am still not convinced of the significance and practical use of the proposed approach in practice, where we want models to perform reasonably well on unseen future data (W5). Therefore, I keep my score unchanged. Thanks.

We thank the reviewer for continuing the discussion. On W5, the practical question we address is how to choose a model when a practitioner has an unlabeled dataset and wants to identify anomalies among its observations. Importantly, SubSelect does not use labels from any of the 39 datasets and uses only the features of the dataset at hand and the candidate models’ anomaly scores to select a model on that dataset. This means that SubSelect is not a meta-learning approach and does not require any information from other datasets. It treats all 39 datasets as unseen data and isolated problems to demonstrate practical utility.

The role of labels in this benchmark of Ma et al. (2023) is to evaluate that choice. If we removed all anomaly labels from the 39 datasets, SubSelect would select exactly the same models. However, what we could no longer do is calculate AP or ROC-AUC to judge how well those models ranked the actual anomalies. The labels are therefore necessary for us to assess selection quality, not for a practitioner to use SubSelect. Evaluating the same procedure on 39 datasets allows us to assess how consistently it selects strong models across a wide range of datasets rather than drawing conclusions from a single application.

Regarding significance, SubSelect achieved the best mean rank on both AP and ROC-AUC among 45 approaches using one fixed configuration across all 39 datasets. It was the only selection method to significantly outperform iForest-r on AP, with mean AP of 0.424 versus 0.399. Pairwise Bayesian signed-rank tests decided 42 of 44 comparisons in SubSelect’s favor on AP and 32 of 44 on ROC-AUC, with none decided against it on either metric. This comparison includes meta-learning methods that can use the other 38 datasets as labeled references, whereas SubSelect uses only the dataset at hand.

A practitioner can apply the same procedure to a new unlabeled dataset without learning from any other dataset. This differs from keeping a model selected on one set of observations and using it to score observations that arrive later, a distinction we have clarified in Section 6. You mention that some of your concerns remain unanswered. Could you clarify what you still find insufficiently supported about SubSelect’s practical use for the unlabeled dataset at hand?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.