- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper selects an anomaly detector without labels. For each candidate model, it takes the top-scored fraction of points and measures how far they lie from the rest of the data, using Mahalanobis distance and a Gaussian KDE fitted on the rest. The candidates are ranked on each measure and the ranks are averaged over 50 top fractions (1% to 50%). On the benchmark of Ma et al. (2023) with 39 datasets and 297 models, it has the best mean AP rank among 37 approaches. It is the only selection method that beats the Isolation Forest baseline (IFOREST-R) significantly on AP, but the gain is small.
Critical strengths
- Simple rule that needs no labels, no reference datasets and no per-dataset tuning.
- Careful statistics (Holm correction, Bayesian signed-rank test, Friedman-Nemenyi) on a public benchmark.
- Thorough ablations (measures, fitting set, fraction grid, pool size) and honest reporting of weak spots.
Critical weaknesses
-
The novelty is modest. Rating a model by how separable its top-ranked points are in feature space is the core idea of IREOS and SIREOS. SubSelect replaces the classifier with MD and KDE and adds rank aggregation over fractions. This is a sensible step, but not a new principle.
-
The contrast measures are themselves standard anomaly detectors, and SubSelect rewards candidates whose top points look anomalous under MD and KDE. It may simply favour candidates that agree with an MD or KDE detector. Please report the AP of plain MD and KDE detectors on all 39 datasets, and the AP of selecting the candidate whose ranking best matches them.
-
The algorithms that get selected are never shown. In high dimension the Scott bandwidth h = |R|^(-1/(d+4)) is close to 1 (0.996 for InternetAds, d = 1,555), so the KDE term should behave like a nearest-neighbour distance. Table 9 fits this: removing kNN hurts most, and removing Isolation Forest raises AP. A table of how often each of the eight algorithms is picked, against the best model per dataset, would settle it.
-
The margins are small. The gain over IFOREST-R is 0.025 AP, and over MC-NDCG 0.020. Against MC-NDCG it wins 18 datasets, ties 2 and loses 19 (Table 10). HITS-auth has the higher mean ROC-AUC (0.785 vs 0.779). The paper should say plainly that it does not beat these two decisively.
-
The datasets are not independent, yet all tests treat the 39 as independent. Pima (ODDS) and Pima (DAMI) have identical n, d and anomaly counts (768, 8, 268), and Annthyroid, Glass, Arrhythmia and the breast-cancer sets also appear twice in some form (Table 3).
-
Assumption A1 (anomalies are rare) fails on a quarter of the data. Ten of 39 datasets have more than 20% anomalies, e.g. Arrhythmia (DAMI) at 45.8%. There, the top subset at small p is mixed and the reference holds many anomalies. Results should be broken down by anomaly rate.
-
Everything is evaluated on the 297 classical models of one benchmark. Section 6 says the method applies to deep detectors, but this is not tested. Related work on hyperparameter optimization and LLM-based model selection for outlier detection is missing [1], [2], [3], and the review of outlier model selection [4] should be discussed.
-
The total cost is not reported. From Appendix A.4.5 I estimate about 121 single-core hours over the 39 datasets (67 for KDE on ALOI). Please give the total over all 297 models and compare it with the consensus methods.
References
[1] Zhao, Y. and Akoglu, L. (2020). Hyperparameter Optimization for Unsupervised Outlier Detection (HPOD). arXiv preprint.
[2] Chen, S. et al. (2024). PyOD 2: A Python Library for Outlier Detection with LLM-powered Model Selection. arXiv preprint.
[3] Yang, T. et al. (2024). AD-LLM: Benchmarking Large Language Models for Anomaly Detection. arXiv preprint.
[4] Ma, Q. et al. (2023). The Need for Unsupervised Outlier Model Selection: A Review and Evaluation of Internal Evaluation Strategies. SIGKDD Explorations, 2023.
Questions for authors
-
How often does SubSelect pick each of the eight algorithms, and how does that compare with the best model per dataset?
-
What AP do plain MD and KDE detectors reach, and what if you select the candidate that best matches them?
-
Do the AP results against IFOREST-R and MC-NDCG survive after removing duplicated sources?
-
How does it perform on the 10 datasets with more than 20% anomalies compared with the other 29?
-
Did you try other bandwidth rules, or a kNN distance as a third measure?
4 comments
Dear Reviewer,
Thank you for your feedback. We would appreciate it if you could confirm before the end of the discussion period whether our responses address your concerns.
Best regards,
The authors
Thanks to the authors for the detailed reply and for running the extra analyses. I could only check it against the submitted PDF, so the new tables are taken on trust for now.
-
Novelty: I still see SubSelect as a sensible step beyond IREOS and SIREOS rather than a new principle, but the design is clearly described and I am fine with how Section 2 positions it.
-
MD and KDE controls: This was my main worry, and the new controls answer it well. If plain MD and KDE, and selecting by agreement with them, both fall well short of SubSelect, then it does more than copy a density detector. Please keep these results in the paper.
-
Selected algorithms: The counts are useful, and kNN is picked much more often than its share of the pool would suggest. How does this compare with the algorithms behind the best models? Since MD alone shows the same preference, I agree my KDE argument does not explain it.
-
Margins and dataset overlap: The revised wording on MC-NDCG and HITS-auth is fair. The ODDS/DAMI split helps, but on DAMI the gain over MC-NDCG is close to zero. It would be good to say this next to the significance claims, and to state that overlapping datasets are counted as independent.
-
Anomaly rate: The breakdown confirms that the method is weaker when anomalies are not rare. Could this go into the limitations section?
-
LLM-based selection: Good to see PyOD 2 and AD-AGENT added. How did they do? The mean ranks quoted in the reply also differ from the submitted Table 1, so I assume the ranking now includes more methods. Is SubSelect still first?
-
Related work: I withdraw my remark on Ma et al. (2023), since the paper is built on that benchmark.
-
Runtime and bandwidth: The totals and the bandwidth check answer my questions.
Overall, the reply is responsive and the new controls make the result more convincing. The novelty is still incremental and the gains are small, so I keep my rating at weak accept.
We thank the reviewer for the re-assessment and detailed reply.
Selected algorithms. Table 7 in Section A.4.4. of the revised manuscript compares in the "Best AP" and "Best ROC-AUC" columns the number of datasets on which each algorithm was best-performing among all 297 models.
Margins and dataset overlap. We will (i) state next to the significance claims in Section 5.2 that the advantage over MC-NDCG is concentrated on the ODDS datasets (mean AP difference 0.002 on DAMI, Table 11), and (ii) state in Section 4.2 explicitly that the tests treat the 39 datasets as independent although some share a common source, which we retain for comparability with Ma et al. (2023).
Anomaly rate. We will mention the weaker results at higher anomaly proportions in the limitations section consistent with the minority assumption (A1), phrased as a descriptive finding.
LLM-based selection. PyOD 2 attains mean AP 0.372 and mean ROC-AUC 0.726, AD-AGENT Selector 0.362 and 0.731 (see Tables 1 and 6 of the revised manuscript). Both comparisons are decided in favor of SubSelect. The quoted mean ranks differ from the submitted Table 1 because the rank pool grew from 37 to 45 approaches. SubSelect is still first: it retains the best mean rank on both metrics with no comparison decided against it.
We will incorporate the proposed changes in the revised manuscript and thank the reviewer for the suggestions that have strengthened the paper.
W1. IREOS and SIREOS
We agree that IREOS and SIREOS are the closest methods. Our contribution is not the general use of input features, but the ranking-based subset comparison. IREOS uses classifier-based separability for individual observations, while SIREOS uses pairwise similarity. SubSelect forms top and reference subsets from each model's ranking, fits measures on the reference and evaluates the top subset. It combines measures and aggregates across top subset fractions. No compared method combines all three components (Section 2). Ablations support this design (Section 5.3).
W2/Q2. MD/KDE controls
To test these simpler alternatives, we added Table 16 (Section A.4.5). MD and KDE fitted and scored on all data give mean AP of 0.3250 and 0.3213. Selecting the candidate whose observation ranking best matches theirs by Spearman correlation gives 0.3509 and 0.3517, against SubSelect's 0.4241. All four also have lower mean ROC-AUC, supporting subset-based selection over these simpler uses of MD and KDE.
W3/Q1. Algorithm selections
The new Table 7 gives SubSelect's selection counts for kNN (14), Isolation Forest (10), LOF (9), OCSVM (3), LODA (2), HBOS (1), ABOD (0) and COF (0). It also gives MD-only/KDE-only counts and algorithms with the best AP/ROC-AUC models.
KDE can act like nearest-neighbor distance when the nearest reference point dominates its estimate. However, MD-only selects kNN more often than KDE-only (14 versus eight datasets), so this preference is not specific to KDE.
W4. Close comparisons
Section 5.2 now reports the mean AP gain over MC-NDCG of 0.020, about 5%, alongside 18 higher, 2 equal, and 19 lower results. SubSelect also has the better mean AP rank (16.01 versus 16.59, Table 1) and is the only compared method to significantly outperform iForest-r on AP. HITS-auth has higher mean ROC-AUC (0.7852 versus 0.7786), while SubSelect has the better mean rank (13.19 versus 16.13, Table 6). Neither Bayesian comparison reaches 0.95 probability in either direction.
W5/Q3. Dataset dependence
You are right that some datasets are not independent, as acknowledged in Section A.3.3. We kept Ma et al.'s models and datasets for comparability and to test for a significant AP gain over iForest-r, which none of their compared methods achieved. Table 11 separates ODDS and DAMI, rather than removing shared sources. Mean AP gains are +0.037 and +0.012 over iForest-r, and +0.035 and +0.002 over MC-NDCG, respectively.
W6/Q4. High anomaly ratios
We agree that datasets that violate A1 are challenging for SubSelect. To mitigate this, SubSelect aggregates across larger subset fractions that accommodate high anomaly ratios. On the 29 datasets with at most 20% anomalies, new Tables 12 and 13 show larger mean AP and ROC-AUC gains over iForest-r. Above 20%, the AP gain becomes much smaller and mean ROC-AUC falls slightly below iForest-r.
W7. Deep models and related work
Section 6 now clarifies that SubSelect can accept deep-model scores, but deep-model selection remains untested. We added hyperparameter optimization and LLM-assisted selection to Section 2 and evaluated PyOD 2 [2] and AD-AGENT Selector. Prior benchmark knowledge from training could inflate LLM performance. This is one reason we initially omitted them.
AD-LLM's dataset-informed selection uses labeled examples, unavailable in our setup [3]. HPOD [1] tunes one algorithm rather than selecting among them. Section 2 discusses it and Ma et al.'s stand-alone and consensus strategies [4].
W8. Runtime
Section A.4.6 now gives total estimates. Scaling one representative candidate's timings to all 297 models and 39 datasets gives 10.5 single-core hours for MD, 110.5 for KDE and 120.9 combined, excluding candidate training and prediction. An application usually concerns one dataset, with a median combined estimate of 12.2 minutes for 297 models and 50 fractions. Models, datasets, fractions and measures can run in parallel.
For consensus methods whose results we reuse, Ma et al. report MC below one hour on ALOI and MCS, Ens and HITS within ten minutes per dataset (their Figure 1). These use model outputs without input features. IREOS, a stand-alone method using input features like SubSelect, was the slowest, taking over 16 days on ALOI. Their timings provide context but were obtained on different hardware.
Q5. Bandwidth and kNN
New Table 15 compares Scott and Silverman rules at multipliers 0.5, 1, and 2. Mean AP ranges from 0.4157 to 0.4253 and ROC-AUC from 0.7730 to 0.7786. Our unchanged Scott default gives 0.4241 AP and 0.7786 ROC-AUC.
We focused on MD and KDE to combine global distance with local density rather than adding kNN's local distance and its choice of k as another parameter. kNN and other contrast measures remain possible additions or alternatives for future work (Section 6).