- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper suggests studying thresholding as a limiting factor in anomaly detection research, show limitations of existing approaches and suggest their own thresholding approach they claim to outperform every other algorithm.
Critical strengths
I agree with the initial idea of studying anomaly-detection components, especially thresholding. Furthermore, the number of approaches considered here is impressive, even though I am still missing one baseline.
Critical weaknesses
I have problems with the reliability of the results. Concretetely I am missing 3 points, that make me question every result reported in the paper.
-
Using only 10 datasets (tabular + image) is very little. Modern benchmarks like https://arxiv.org/abs/2602.09329 suggest using orders of magnitude more datasets, yet remain susceptible to noise from datasets that prefer different algorithms. As you benchmark more methods! Using less than 1% of the datasets suggested makes each of your results untrustworthy. I know that there might be computational limitations, but trustworthy results require at least an order more datasets.
-
You are missing one baseline. In 3.2, you study the 3S rule and accurately criticize it for potentially treating all anomalies as normal. However, there is a simple baseline that you missed (as far as I can see): Simply considering, e.g., 99% of the training data as normal and using this to define a threshold virtually guarantees a valid threshold and requires negligible computational cost and is the default in the largest library used for anomaly detection PyOD. Missing this baseline is a critical mistake and reason enough to reject your paper. Please add it. If I simply missed it during reading (your naming is a bit confusing), please point out what you call it.
-
Your anomaly detection methods are slightly outdated, especially with regard to thresholding difficulties. Namely, you are missing modern tabular foundation models (e.g., https://arxiv.org/abs/2603.19497 or https://arxiv.org/abs/2409.05672v3 or https://arxiv.org/abs/2602.03018). These are not only more powerful than the outdated methods you choose, but are also often trained to output binary labels, which affects thresholding choices. E.g., the threshold 0.5 is a valid choice. Whether thresholding methods can improve this further or whether modern foundation models make thresholding a question of the past would be interesting in either case.
Further, your presentation requires work. In particular, your Figures tend to be very confusing and lack the information needed to understand them. For example, in Figure 3: what is the x-axis? Normalizations? Then why no entry "no normalization"? Also which datasets? Tabular or Timeseries? I would also have liked a description of each of the 38 methods in the main paper, or at least of their general ideas. Your paper works partially as a benchmarking paper on thresholding, but I did not learn anything about the approaches people use for thresholding from the main paper. And the descriptions in the appendix are not much better: You mention the methods, but there is very little deeper understanding shown here. The statement in the abstract that over half of studies and libraries require supervision would be very strong, but requires justification that I missed in the paper.
Questions for authors
In its current form, I cannot recommend accepting the paper. However, if you could increase the quality of the reported results (Points 1-3 under weaknesses), I would change my recommendation to either weak or clear acceptance.