- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper points out that in industrial control systems, the metrics of anomaly detectors cannot tell which component of the pipeline leads to poor performance. For this, the authors decompose the anomaly detector to three stages and propose a workflow based on covariance-normalized residual energy to do the controlled comparisons of the normal reference, the residual-producing model, and the scoring and alarm rules. Experiments on 5 detectors and 3 datasets show that the effects of reference updates and rescoring vary across detector–dataset pairs. A GDN case study further shows that the effect of a model change depends on how the reference is fitted.
Critical strengths
- This paper clearly separates the interventions by the pipeline stages, and designs targeted controlled experiments to evaluate the effects of these interventions.
- The GDN case study compares the results of the same intervention under different reference protocols, indicating that the measured effect can reverse. This highlights that a model change cannot be interpreted without specifying how the normal reference is fitted.
Critical weaknesses
-
The related work is not comprehensive. Only 3 of 23 references are from the latest 3 years (2024 ~ 2026). Some representative work are not discussed. For example, [1] proposes to determine which sensor or actuator has been manipulated by applying attribution methods to anomaly detectors. [2] has already used the methodology that keeps the trained model unchanged and separately studies the downstream decision rules. Besides, it has discussed the normal training-test distribution differences of SWaT and WADI.
-
The difference from some related work is not stated clearly. Although the authors cites Garg et al.(2022)[3], they have not convincingly established the additional diagnostic value of the proposed workflow. [3] decomposes the detector to a model, a scoring function and a thresholding function, and varies the model and the scoring function independently on a grid of 10 models and 4 scoring functions. The "evidence extraction" and the "controlled comparison" principle in this paper are basically extensions of it. Besides, it also deploys a sliding-window update and observes that such scoring functions "can also adapt to anomalies in the test set". The authors should state what is new precisely.
-
The policy comparison is retrospective and can be contaminated by attacks. The D_ref is computed based on the whole evaluation recording, and then the policy is evaluated on the same recording. The included "attack periods" can increase the statistics. A prospective screen computed on a trailing window is necessary for the practical value, not just an optional extension.
[1] Fung C, Zeng E, Bauer L. Attributions for ML-based ICS Anomaly Detection: From Theory to Practice[C]//NDSS. 2024.
[2] Fung C, Srinarasi S, Lucas K, et al. Perspectives from a comprehensive evaluation of reconstruction-based anomaly detection in industrial control systems[C]//European Symposium on Research in Computer Security. Cham: Springer Nature Switzerland, 2022: 493-513.
[3] Garg A, Zhang W, Samaran J, et al. An evaluation of anomaly detection and diagnosis in multivariate time series[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33(6): 2508-2517.
Questions for authors
- Could you elaborate on the related work and explain clearly the differences between this paper and the related work?
- Could you add the experiment for the prospective screen on the same recording?
- In Appendix A, it is stated that TranAD uses the v3 residual archive. Could you explain what "version" means and why it is v3? Does the choice affect the results?
2 comments
Thank you for the response and experiments. The revised version addresses my questions. Weakness 1 has been resolved, and Weaknesses 2 and 3 have been explained and supplemented with experiments. However, I maintain my rating because the paper's contributions as a method and as an analysis are still limited. As a method, the causal trailing-window screen only matches always updating, and Section 5.2 also states that the reference screen does not improve on simpler drift checks. As an analysis, the revision validates the reference diagnosis by intervening on specific input channels, but the more general conclusions, such as benign test data differing from benign training data and adaptive scoring absorbing sustained attacks, are close to the existing findings of Fung et al. (2022) and Garg et al. (2022).
Thank you for the review and for the references.
W1 and Q1, related work. We added both papers to Section 2. Fung et al. (2022) keep the trained model fixed while they study the detection rule, and they observe that some benign test data in SWaT and WADI differs from the benign training data and remove features with a Kolmogorov-Smirnov test. Fung et al. (2024) attribute detected attacks to manipulated sensors. Our revision relates to both. We locate the difference between training and test in time and in specific channels, we test it by intervention, and our attribution targets the elevation of normal residual energy, not the attacked component.
W2, what is new relative to Garg et al. We agree that evidence extraction and controlled comparison follow their decomposition, and that they already observed that adaptive scoring can absorb anomalies. We state three differences. First, they vary the model and the scoring function. We treat the normal reference as a third component that is controlled separately. Second, we test the reference diagnosis by an intervention on the detector's inputs (Section 5.6). Third, we measure what the reference changes in comparisons between detectors. Across four label-free reference protocols, 36 of 540 comparisons between two detectors reverse with intervals that exclude zero (Appendix M). We also compared with their Gauss-D scoring directly (Appendix I).
W3 and Q2, prospective screen and contamination by attacks. We agree that this is necessary and ran it on the same recordings (Appendix H). A trailing median over 24 hourly blocks makes the same seven update decisions with no false trigger. It reacts about 12 hours late, and its mean AUC of 0.756 is no better than always updating at 0.759. Shorter windows react sooner but trigger on the ten-hour SWaT attack. Boundaries calibrated on training data alone were too tight and triggered falsely on most retained pairs. The revised abstract and Section 5.2 report that the causal version only matches always updating.
On contamination, we recomputed the screen without the attack timestamps and without the longest run. All fifteen decisions are unchanged (Appendix G). The statistic itself does move. For TranAD on SWaT it doubles when attacks are removed, but it stays far above the boundary.
Q3, TranAD v3. The version refers to our residual archive, not to the method. The SWaT model in our original archive had collapsed to a constant output, with a training error of 0.139 against 0.0068 after the rebuild, and no checkpoint had been kept. Version 3 rebuilds all three datasets with the same preprocessing and recipe and keeps the checkpoints. On SWaT it uses a learning rate of 0.001 instead of 0.008 and the seed with the lowest training loss, and no test labels enter this choice. On WADI and HAI the original archives were valid, and the choice does not affect the results. Fixed-reference AUC is 0.611 against 0.612 on WADI and 0.917 against 0.909 on HAI (Appendix P).