- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes an anomaly detector that is a combination of two anomaly detectors: one based on some reconstruction error from a transformer, and the other based a Mahalanobis distance to non-anomaly data distribution in the latent representation. These two measures are combined using the joint CDF to obtain a final anomaly probability score, which is a smart idea, because the two anomaly scores have inherently different scales.
Although it is applied to a Transformer-style encoder-decoder timeseries model, it is trivially generalizable to any encoder-decoder model.
Critical strengths
The idea is clear and sound. Combining different scores into one is always problematic, and using the 2D CDF is a very nice solution for this.
Critical weaknesses
I'm not sure if it is a critical weakness, but this approach requires that the training data is relatively clean. For both training the model, and fitting the Mahalanobis mean+covariance, you cannot have too much pollution of outlier data. When the Mahalanobis distance is used in high-dimensional spaces (say, more than 100D?) it becomes more important to have enough clean target data.
Questions for authors
To honor past research, the idea of combining different anomaly scores using the 2D CDF is not new. So it would be nice to cite: Li, Z., Zhao, Y., Hu, X., Botta, N., Ionescu, C., & Chen, G. H. (2022). Ecod: Unsupervised outlier detection using empirical cumulative distribution functions. IEEE Transactions on Knowledge and Data Engineering, 35(12), 12181-12193.
Next, could the authors give an idea on how sensitive this approach is to the fraction of anomalies in the training data. In particular, how sensitive is the Mahalanobis distance to additional outliers?
3 comments
Dear authors, Thank you for your additional experiments. I think that (in contrast to "the standard unsupervised TSAD setting") assuming clean training data is not very realistic in practice, and I'm therefore happy to see you used a robust estimator. Thanks.
Additional Response to 2TRM
Dear Reviewer 2TRM,
Thank you very much for suggesting an important experiment and for taking the time to review our manuscript. We are glad that our additional experiments have helped address your concerns about the practical applicability of our approach.
We hope our work will inspire further research toward practical time-series anomaly detection.
If you have any further questions, we would be happy to hear them.
Best regards,
Authors
Response to Reviewer 2TRM
Dear Reviewer 2TRM,
We sincerely thank you for your helpful feedback and insightful comments. In what follows, we address your concerns one by one. Please refer to the revised manuscript, as all our responses follow its updated content.
[Response to W1-1, Q2] Thank you for this insightful question. As stated in the Introduction, we would first like to clarify that our paper follows the standard unsupervised TSAD setting that assumes normal training data. However, as you pointed out, evaluating robustness to anomalies in training data is also important. To this end, we conduct additional experiments to examine the sensitivity of our approach to training data contamination.
Specifically, we evaluate training contamination ratios of 0%, 1%, 5%, 10%, and 30% by adding Gaussian noise with a magnitude of three times the training standard deviation to selected training windows. As shown below, JELI maintains strong anomaly detection performance even at 30% contamination, achieving a VUS-PR of 0.5742 and a VUS-ROC of 0.9006. Notably, these results still outperform all baselines evaluated without additional training contamination (see Table 1), demonstrating that our approach maintains strong performance even under substantial training contamination. (Note that we report results in terms of VUS-PR / VUS-ROC)
The Mahalanobis-based latent-space score also maintains stable performance under contamination, achieving a VUS-PR of 0.5143 and a VUS-ROC of 0.8752 at 30% contamination. By modeling the normal distribution based on semantic information captured in learned latent representations, this design can help reduce the influence of outliers in the training data [1]. The observed stability of latent-space scoring supports this interpretation. We have incorporated these experimental results and the corresponding discussion into the revised manuscript (Appendix A.2).
[Response to W1-2] Thank you for this insightful comment. We agree that reliable covariance estimation is important when using Mahalanobis distance in high-dimensional latent spaces. We have considered this issue in our design and adopted the Ledoit-Wolf shrinkage estimator [2], as described in Appendix A.1. This estimator regularizes the covariance matrix to improve numerical stability, allowing Mahalanobis distance to be computed even when the number of reference samples is smaller than the latent dimension. This addresses the concern regarding high-dimensional covariance estimation, while sensitivity to anomalous training samples is examined separately in our response to [W1-1, Q2].
[Response to Q1] Thank you for suggesting this valuable reference. We agree that ECOD [3] is an important prior work on CDF-based anomaly scoring. Specifically, our approach models the joint distribution of complementary anomaly evidence from the input and latent spaces, whereas ECOD computes empirical CDFs for individual input features and combines their tail probabilities. We believe this is an important methodological distinction and have added the suggested reference along with a clarification in the revised manuscript (Section 2).
References
[1] Assran et al., Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture, CVPR 2023
[2] Ledoit et al., A Well-Conditioned Estimator for Large-Dimensional Covariance Matrices, Journal of Multivariate Analysis 2004
[3] Li et al., ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functions, IEEE Transactions on Knowledge and Data Engineering 2023
We hope that our additional experiments and explanations have addressed your concerns and provided further evidence supporting our framework. If you have any further questions or would like to discuss any remaining points, we would be glad to continue the discussion.
Thank you for your valuable time.
Best regards,
Authors