- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper learns, for each time series, a weight vector over 12 fixed anomaly detectors, and scores the series with the weighted sum of the detector scores. On TSB-UAD it compares three ways of learning the weights: gradient descent on an MSE proxy (ML), PPO with a VUS-PR reward (RL), and a genetic algorithm (GP). In-distribution, RL on catch22 features reaches a mean VUS-PR of 0.440, above the best model selector (0.382) and below the per-series oracle (0.489). ML and GP fall below the model selector.
Critical strengths
- Clear, well-posed question; the per-series Oracle is a useful ceiling for model selection.
- One pipeline (data, splits, detectors, network) for all three learning methods.
- Both in-distribution and leave-one-dataset-out settings are tested, and failures are reported.
- The per-series analysis of where the ensemble beats the Oracle (Fig. 5) is informative.
Critical weaknesses
-
The main conclusion does not follow from the design. The paper credits the RL win to its "direct optimization of the true TSAD metric", but RL also differs from ML and GP in loss, training budget and tuning. Nothing isolates the learning paradigm as the cause.
-
The budgets are unmatched. RL sees 720,000 states, while GP runs 20 generations of 20 solutions on 256 series each (about 100K series evaluations) over a much larger search space. Training takes about 25 h for RL and 5 h for GP (Fig. 6a). A fair comparison should match the number of metric evaluations or the wall-clock time.
-
The ML proxy is weak. MSE against rare 0/1 labels mostly rewards low scores on normal points, while VUS-PR measures ranking. Ranking surrogates for AUC-PR exist (e.g. [1], [2]), as do pair-wise losses for weighting anomaly detectors [10], and they should be tried. Otherwise "ML" simply means "MSE".
-
A baseline with a ranking (pair-wise) score function should be used for the classical ML method. Work on supervised ensembles of anomaly detectors [10] shows that learning detector weights from labels improves performance, and that a pair-wise comparison loss (an anomaly should score above a nominal) improves accuracy at the top of the ranking, which is also what the anomaly detection metric measures. The ML baseline here instead uses MSE against the 0/1 labels. A pair-wise or other ranking loss on the same weighted-sum model should be added, so that the comparison with RL and GP does not rest on a weak loss.
-
The RL here is a one-step contextual bandit ("each state-action-reward tuple is treated independently"); PPO reduces to a score-function gradient with a learned baseline. Calling it a separate "paradigm" overstates the difference. Policy gradients for non-differentiable metrics are standard (e.g. self-critical training [8]), as are evolution strategies [9]; the RL and GP choices should be placed in this context.
-
Simple baselines are missing: (a) one global weight vector learned on the training set, (b) score-level stacking on the 12 detector scores (as in XGBOD [3]), (c) average of the top-k detectors chosen by the model selector. The paper rules out stacking explicitly, but the title's claim needs at least (a) and (b). Related work on supervised and selective outlier ensembles (XGBOD [3], LSCP [4], SELECT [5], MetaOD [6]) is missing, as is the review of model selection for outlier detection [7].
-
Variance is understated. The five seeds seem to vary only the training, not the 70/30 split. Figs. 7a and 7c use a single seed. It is also not said whether the Best MS value was re-run with the same VUS-PR code or copied from the earlier paper.
-
Several details are missing: the number of series per split, the score normalization of the 12 detectors, what happens when all clipped weights are zero, and how ML/GP outputs are kept in [0, 1] (Sec. 4 says [0, 1], Appendix A.9 says [-1, 1]).
References
[1] Qi, Q., Luo, Y., Xu, Z., Ji, S. and Yang, T. (2021). Stochastic Optimization of Areas Under Precision-Recall Curves with Provable Convergence. NeurIPS 2021.
[2] Brown, A., Xie, W., Kalogeiton, V. and Zisserman, A. (2020). Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval. ECCV 2020.
[3] Zhao, Y. and Hryniewicki, M. K. (2018). XGBOD: Improving Supervised Outlier Detection with Unsupervised Representation Learning. IJCNN 2018.
[4] Zhao, Y., Nasrullah, Z., Hryniewicki, M. K. and Li, Z. (2019). LSCP: Locally Selective Combination in Parallel Outlier Ensembles. SDM 2019.
[5] Rayana, S. and Akoglu, L. (2016). Less is More: Building Selective Anomaly Ensembles. ACM TKDD, 2016.
[6] Zhao, Y., Rossi, R. A. and Akoglu, L. (2021). Automatic Unsupervised Outlier Model Selection (MetaOD). NeurIPS 2021.
[7] Ma, Q. et al. (2023). The Need for Unsupervised Outlier Model Selection: A Review and Evaluation of Internal Evaluation Strategies. SIGKDD Explorations, 2023.
[8] Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J. and Goel, V. (2017). Self-critical Sequence Training for Image Captioning. CVPR 2017.
[9] Salimans, T., Ho, J., Chen, X., Sidor, S. and Sutskever, I. (2017). Evolution Strategies as a Scalable Alternative to Reinforcement Learning. arXiv:1703.03864.
[10] Das, S., Wong, W.-K., Dietterich, T., Fern, A. and Emmott, A. (2020). Discovering Anomalies by Incorporating Feedback from an Expert. ACM Transactions on Knowledge Discovery from Data (TKDD), 14(4), 1-32.
Questions for authors
-
Does the RL advantage remain if ML uses a ranking surrogate loss for AUC-PR, such as a pair-wise loss [10]?
-
How do the three methods compare under matched budgets?
-
Where does D_val come from? Do the seeds resample the 70/30 split?