- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates unsupervised online anomaly detection in spectral data streams under varying vacuum conditions. The authors extend existing benchmarks by introducing BV5, which includes new degassing cycle scenarios, different leak types, and additional sensor measurements. Through a comparative evaluation of existing online anomaly detectors, the study highlights their varying robustness under complex operating conditions. Additionally, the authors propose a Subset Framework (SF) that filters unreliable spectral observations using auxiliary sensor information, demonstrating its potential to improve anomaly detection performance.
Critical strengths
- Practically relevant benchmark extension. The paper introduces newly collected spectral datasets covering degassing cycles, different leak types, and varying operating conditions. These scenarios extend previous benchmarks and provide a useful resource for evaluating anomaly detection methods under more realistic industrial conditions.
Critical weaknesses
-
Limited novelty and insights beyond existing benchmarks. The paper primarily extends the spectral anomaly detection benchmarks established in previous work, particularly Varela et al. (AAAI 2026). While the new scenarios are practically relevant, the authors do not sufficiently demonstrate what fundamentally new challenges or insights they introduce. The cumulative comparisons mainly reveal changes in algorithm rankings, without systematically explaining the underlying failure mechanisms or identifying new research directions. Consequently, the incremental contribution may be insufficient for a standalone benchmark study.
-
Insufficiently controlled evaluation of the proposed Subset Framework. The experiments comparing SF with the original detectors use different hyperparameter configurations, including substantially different window sizes and ensemble sizes. Therefore, the reported performance improvements cannot be reliably attributed to the proposed filtering strategy alone. Controlled comparisons under matched hyperparameters are necessary to establish the actual contribution of SF.
-
Unclear effects of data filtering on detection performance. SF excludes a substantial proportion of spectral observations and directly assigns zero anomaly scores to excluded instances. This raises concerns about whether the improvements reflect better anomaly detection or simply a restriction of the effective detection space. The paper lacks comparisons with simple rule-based or random filtering strategies, as well as analyses of anomaly coverage and missed detections. These omissions weaken the evidence supporting SF's effectiveness.
-
Incomplete benchmark coverage and evaluation. Although the paper evaluates several traditional online anomaly detectors, the coverage of recent deep learning approaches remains limited, with KitNet being the only evaluated deep learning method. Relevant approaches such as ARCUS are discussed but not included. Furthermore, despite emphasizing real-time anomaly detection, the evaluation primarily focuses on VUS-PR without systematically assessing computational efficiency, memory consumption, or detection latency. This limits the comprehensiveness and practical value of the benchmark.
-
Limited dataset diversity and reproducibility. The newly introduced datasets are collected from a single controlled experimental setup, with fixed leak intensity and relatively few abnormal events in several scenarios. Consequently, their representativeness across different industrial systems and operating conditions remains unclear.
Questions for authors
-
Reliability of anomaly annotations. According to Appendix B, anomaly labels are primarily determined by the timing of leak valve operations. However, the actual spectral response may not align precisely with valve opening and closing events. How do the authors account for potential response delays, and how is the accuracy of the ground-truth annotations validated?
-
Hyperparameter tuning protocol. The hyperparameters are selected using AUC-PR on only two labeled datasets, including one spectral dataset from BV4. Could the authors clarify how this supervised tuning procedure aligns with the intended unsupervised setting? How sensitive are the conclusions to the choice of tuning datasets, particularly given the different operating conditions in BV5?