- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes TOAD, a method for unsupervised anomaly detection and root-cause analysis in multivariate time series. For each window, it builds a Pearson correlation graph, encodes the window with a GCN and a temporal convolution, and pools everything into one embedding. The anomaly score combines three terms: a sliced Wasserstein distance between recent embeddings and regime-specific normal references, a reconstruction error, and a temporal-transition term. Sensor-level root causes are obtained from the gradient of this fused score, so the explanation is tied to the detector's own decision. The authors also give Lipschitz and concentration results that claim stable scores and stable attributions. Experiments on SMD, SMAP, MSL, PSM and SWaT show the best VUS-ROC on four datasets, better hit rates on synthetic sensor injections in SMD, and a counterfactual repair test for faithfulness.
Critical strengths
- The paper targets a real gap: anomalies that change cross-sensor dependencies while each sensor stays in its normal range. These are hard to catch with pointwise reconstruction or forecasting errors and deserve more attention.
- Tying the explanation to the same score that raises the alarm is the right idea for operators. The paper also keeps faithfulness separate from causal correctness, which is a fair and useful distinction.
- Regime-specific references with a minimum over regimes is a simple and practical way to avoid false alarms when the system moves between normal operating modes. Many detectors ignore this.
- Fusing transport, reconstruction and temporal evidence, with robust normalization fitted on training data only, is a clean design. It keeps each stream on a comparable scale and avoids using test statistics for normalization.
- Using counterfactual repair to test whether attributed sensors actually drive the score is a good evaluation idea in principle. It checks the explanation against the model's decision instead of only against proxy labels.
Critical weaknesses
- The RCA method is not clearly defined. TOAD's own gradient attribution (21.3 in Table 5) is worse than GDN (23.5) and TranAD (25.8) at reducing TOAD's own score, so the gain of "TOAD RCA" comes from steps that are never described.
- The theory does not match the model. The proofs assume a fixed graph, independent windows and smooth activations, but TOAD recomputes the graph per window, uses overlapping stride-1 windows and LeakyReLU.
- Some proofs are wrong even under their own assumptions. Theorem 1 says the false alarm bound improves with d and T, but X_max bounds the Frobenius norm of a window and grows with sqrt(dT) for bounded sensor values, so L_f · X_max = K · x̄ and the d and T terms cancel. Theorem 2(iii) bounds the probability over the random draw of training windows while the test input is fixed. So it only says attributions concentrate around their mean, not that a wrong sensor is unlikely to be flagged during deployment.
- Hyperparameters seem to be tuned on test data. The chosen values in Figure 4 are at the peak for all five datasets, and no validation split is mentioned.
- Some baseline numbers look odd. All Table 1 means for DADA, GDN, KAN-AD and KMeansAD are exact integers, and PCA on SMD has 83.1 AUC-ROC but 39.9 VUS-ROC.
- RCA is tested only on synthetic data, although real labels exist. SMD has interpretation labels, used by OmniAnomaly and TranAD to report H@P%, and the SWaT attack list names the attacked sensors.
- The value of OT is not shown. With stride 1 and W=10 the windows almost fully overlap, and W=1 loses only about 1.5 points, so a kNN baseline on the same embeddings and a comparison with MADGA are needed.
- Gains are small on VUS-PR, the metric TSB-AD recommends. On SMD (+0.2) and SMAP (+0.3) they are within one standard deviation.
- The method is described inconsistently. Figure 1 and Eq. 6 list different losses, and L_reg is always zero after hard projection, yet removing it changes results.
- The dataset claims are weak. SMAP and MSL are mostly binary command channels where Pearson graphs are undefined, and SWaT attacks are not mainly slow single-sensor drifts.
Questions for authors
- What exactly is "TOAD RCA" in Tables 4 and 5? Please give the full algorithm, including the counterfactual term and graph-aware propagation, since Sec. 3 only defines a gradient norm.
- Why does TOAD's own gradient attribution reduce TOAD's score less than GDN and TranAD rankings in Table 5?
- Can the theorems be corrected for a per-window graph, overlapping windows and LeakyReLU? Which guarantees still hold for the evaluated model?
- Does the counterfactual step inside TOAD RCA use the same repair operator as the faithfulness test? If yes, how is the test not circular?
- How were T, k, R, W and the fusion weights chosen? Was Figure 4 computed on test data, and if so, what are the results with a separate validation split?
- Can you share per-seed results or logs for the baselines in Table 1? Why are all means for DADA, GDN, KAN-AD and KMeansAD exact integers, and why does PCA on SMD have 83.1 AUC-ROC but 39.9 VUS-ROC?
- Why was RCA not evaluated on the real SMD interpretation labels or the SWaT attack list? What are H@P% and N@P% on these?
- Which training objective produced the reported results, the one in Figure 1 or Eq. 6?
- With hard spectral projection after every step, L_reg is always zero. What does "w/o spectral reg." actually remove?
- How are constant channels handled in the Pearson graph, especially on SMAP and MSL where most channels are binary command flags?
- In the faithfulness test, windows are picked by TOAD's score and all methods are judged under TOAD's threshold. How do the results change if windows are picked by ground-truth labels?
- TSB-AD recommends VUS-PR as the most reliable metric, so why did you rank methods by VUS-ROC? Under VUS-PR (Table 6), the gains on SMD (+0.2) and SMAP (+0.3) are within one standard deviation. Do you still claim state-of-the-art detection on these datasets?
Suggestions
- Please compare against a kNN or Mahalanobis score on the same embeddings, and against MADGA, to show what OT adds.