- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
-
This paper investigates signed-orientation response in trained time-series anomaly detection (TSAD) scores, particularly whether models with similar native AUC exhibit different responses to the same sign-flip transformation. The authors introduce a partial channel sign flip that preserves every coordinatewise magnitude while reversing signed products across the flipped/unflipped channel partition. Response strength is measured through rank separation between original and flipped normal windows.
-
An exact even–odd decomposition separates the same trained score into flip-invariant and flip-sensitive parts. The authors compare the full score with its invariant part on labeled natural data to examine whether the flip-sensitive component helps or harms detection, without retraining.
-
Nine released TSAD implementations are evaluated across 65 multivariate views. A separate common panel of 56 views from six datasets supports paired natural-detection and training-budget comparisons. At 50 epochs, removing the flip-sensitive component changes magnitude-matched AUC by more than one percentage point in 19 of 54 model–dataset combinations, with effects in both directions. The experiments also show that similar native AUCs can coexist with large differences in response strength.
Critical strengths
1. Careful empirical protocol.
- Nine publicly released TSAD models are evaluated under a common preprocessing and evaluation protocol, with frozen code revisions and hierarchical-bootstrap confidence intervals.
- The high-response subgroup and magnitude matching were specified before examining the relevant natural-detection results.
2. Connection to natural detection.
- The exact even-odd decomposition isolates the flip-sensitive component of the same trained score without retraining. Removing it changes magnitude-matched AUC by more than one percentage point in 19 of 54 model-dataset pairs at 50 epochs. It shows that response strength and detection utility are distinct.
Critical weaknesses
1. Unclear task-level significance and added value
-
With flipped windows as the positive class, measures rank separation between original and flipped normal windows, that is, how well a score detects one synthetic perturbation of normal data. That models with similar native AUC respond differently to this flip is not surprising: Anomaly Transformer and GDN differ by 1.00 native-AUC point but by 11.36 points in response strength (Abstract; Table 9). The paper does not show whether this response corresponds to natural anomalies in the evaluated datasets, or what it adds beyond the behavioral-testing and underspecification work it cites (§2; Ribeiro et al., 2020; D'Amour et al., 2022).
-
Larger responses tend to accompany larger natural changes (descriptive across nine model averages at 50 epochs; Figure 3b; Table 11). However, this association concerns the size of natural detection changes, not whether the flip-sensitive component helps or hurts detection. The flip-sensitive component changes Anomaly Transformer's magnitude-matched AUC by +14.4 points on HAI and −3.1 on PSM at 50 epochs (Figure 3a). Without labels, the audit cannot determine whether a stronger response is desirable; with labels, is a more direct measure. The added evaluation or deployment value of therefore remains unclear.
2. Limitations of the response metric () and incomplete attribution
-
summarizes rank separation on normal windows, whereas the odd score component gives the pointwise flip-sensitive contribution (§3.3). DCdetector has at both budgets, yet removing its flip-sensitive component changes magnitude-matched AUC by about 1.9 points. This is comparable to TimesNet at 5 epochs (, 1.8 points; Table 11). The USAD–DCdetector contrast in §6 therefore characterizes rank separation, not whether DCdetector's flip-sensitive component is negligible.
-
Appendix B gives no quantitative results for the signed-marginal audit or the relational-shuffle control, and its focused sanity checks use constructed scorers rather than the nine released models. is also reported only for the canonical group (§4; §5.2), whereas Appendix L examines channel-group sensitivity for alone.
Questions for authors
Q1. Task-level significance (Weakness 1)
- Do the labeled anomalies in the evaluated datasets (e.g., HAI or SWaT attacks) exhibit changes in signed cross-channel relations that the sign-flip audit is intended to probe?
- Since the sign of varies across datasets, how should be used when labels are unavailable, and what does it add beyond the behavioral-testing and underspecification work cited in §2?
Q2. Interpretation of low response strength (Weakness 2)
- DCdetector has near-chance response strength (about 0.01), yet removing its flip-sensitive component changes magnitude-matched AUC by about 1.9 points (Table 11). How does this affect the USAD–DCdetector contrast in §6? Would the authors report a paired sensitivity measure, such as the mean absolute odd-score contribution normalized by the score spread on normal windows, alongside ?
Q3. Attribution and robustness (Weakness 2)
- Could the authors report the quantitative results of the signed-marginal audit and relational-shuffle control (Appendix B), ideally for the nine released models? Do the size and sign pattern of hold under the alternative channel groups examined in Appendix L?
3 comments
Attribution controls and channel-group robustness
3. Quantitative attribution controls (W2/Q3)
We added independent-channel shuffling to the released-model evaluation. This preserves each channel's collection of normal-window trajectories while disrupting their cross-channel correspondence. A synchronous shuffle applies the same permutation to every channel as a control. Independent shuffling uses three fixed repetitions.
Results below use the 56-view common panel at 50 epochs, with Time-RCD evaluated zero-shot.
| Model | Original | Synchronous shuffle | Independent shuffle |
|---|---|---|---|
| Anomaly Transformer | 0.3523 | 0.3523 | 0.2456 |
| GDN | 0.2661 | 0.2661 | 0.2528 |
| ModernTCN | 0.1837 | 0.1837 | 0.0065 |
| TimesNet | 0.2547 | 0.2547 | 0.0071 |
| USAD | 0.2182 | 0.2182 | 0.2520 |
| ScatterAD | 0.0282 | 0.0314 | 0.0190 |
| Time-RCD | 0.0032 | 0.0032 | 0.0056 |
Independent shuffling reduces response strength by 97.2% for TimesNet, 96.5% for ModernTCN, and 30.3% for Anomaly Transformer. GDN and USAD retain high separation, demonstrating that the effect of disrupting channel correspondence differs across models.
We complemented this test with four-state scoring of , , , and . We compare
\delta_2=s(-x)-s(T_{\bar G}x).$$ Opposite signs cannot occur for $s(x)=h(f(x_G)+g(x_{\bar G}))$ with strictly increasing $h$. Using numerical screening and requiring both effects to exceed 1% of the normal-score standard deviation, strict reversals occur in **84.74% of normal windows for Anomaly Transformer, 66.57% for GDN, 57.43% for ModernTCN, and 46.46% for USAD**. Thus, the models retaining high separation after shuffling can still exhibit cross-group interaction. For context-dependent models, including TimesNet, ScatterAD, and Time-RCD, we retain and identify the original scoring context rather than claim that all other windows were held fixed. These deterministic controls cover seven models: DCdetector is excluded for the readout issue described in Part 1, and CATCH because repeated evaluation is stochastic. We also quantified a training-fitted independent-channel ECDF scorer. Its maximum view-level mean absolute odd component is **$5.43\times10^{-13}$**. We will report score differences alongside rank-based measurements, since ranking tiny floating-point differences can otherwise give misleading separation values. **4. Does the natural-detection effect hold under alternative groups? (Q3)** We completed 1,064 model/view/seed tasks, evaluating ten channel-group configurations per task. These include top-1/2/4, the canonical group, three random groups, a low-correlation group, and groups containing 25% or 50% of the channels. The trained score, preprocessing, evaluation windows, and original matched sets are held fixed. Model response rankings correlate with the canonical ranking at **0.964–1.000** for the alternatives except the low-correlation group (**0.607**). The direction of $D$, however, is more group-dependent: | Alternative group | Cells with the same sign of $D$ | Opposite signs, with both magnitudes above 1 pp | | :--- | :---: | :---: | | Top-4 | 33/42 | 2 | | Top-25% | 35/42 | 1 | | Top-50% | 27/42 | 3 | For example, AT–HAI remains strongly positive under top-4 (**+13.75 points**, versus **+14.37** for the canonical group), whereas AT–PSM changes from **−3.15 to +1.20 points**. We will include these results explicitly: response rankings are broadly stable across these groups, but the natural-detection benefit of the odd component is not independent of the chosen group. The revision will add the paired-sensitivity measure, quantitative attribution controls, and group-wise natural-detection results, and correct the DCdetector comparisons. We would appreciate your assessment of whether these additions address the metric and attribution concerns, and whether the validation-based application clarifies the practical value.I appreciate the authors' substantial additional experiments and clarifications, which address several of my concerns.
However, my central concern regarding task-level significance remains. The connection between the audited behavior and natural anomalies is still limited, and the proposed measure does not establish whether the observed sensitivity is beneficial for anomaly detection. The group-dependent detection effects further complicate its practical interpretation.
Therefore, I maintain my original rating.
Practical value and paired sensitivity
Thank you for recognizing the empirical protocol and the connection to natural detection. Your distinction between distributional rank separation and pointwise sensitivity is important. We have added measurements addressing that distinction, practical use, and the missing attribution controls.
1. Natural relevance and the role of the audit (W1/Q1)
To examine natural anomalies, we measured cross-group signed relations relative to a training-normal reference on the original magnitude-matched windows. The resulting relation score has AUC 0.7102 on PSM, 0.3941 on HAI, and 0.5109 on SWaT. We retain the original direction of the statistic rather than reversing scores below 0.5.
As a temporal-alignment control, we circularly shifted the complementary channels within each window. Across the 49 nonzero shifts, median AUCs are 0.4893 for PSM, 0.5847 for HAI, and 0.4892 for SWaT. These results identify label-associated relation differences in some datasets, rather than a uniform pattern across all datasets.
Within the 11 HAI views where the statistic is defined, its correlation with signed Dmatched is ρ=−0.845 for Anomaly Transformer and +0.736 for TimesNet. We will report these as descriptive associations, not as evidence that natural attacks reproduce the exact sign-flip intervention.
We agree that larger CT does not establish that the response is desirable. Without anomaly labels, its role is to measure a particular behavioral difference between trained scores, including scores with similar native AUC. When labels are available, D directly evaluates the effect of symmetrization; the two quantities answer different questions.
We also tested a concrete use of the decomposition: selecting between s and seven using validation labels. For Anomaly Transformer, this improves held-out native AUC by 1.11 percentage points (95% CI [0.24, 2.17]) and matched AUC by 0.95 points ([0.05, 2.04]), relative to always using the original score. A broader model-then-variant selection analysis shows little additional gain, so we present this as a model-specific application.
Relative to the behavioral-testing and underspecification work cited in Section 2, our contribution is the specific magnitude-preserving intervention, its exact decomposition of the same trained score, and the empirical connection between this decomposition and natural detection. We will make this distinction explicit rather than present behavioral heterogeneity itself as the novelty.
2. Low response strength, paired sensitivity, and DCdetector (W2/Q2)
We agree that low CT does not imply a negligible pointwise odd component. We now report the requested paired measure on normal windows:
PT=SD[s(X)]E∣sodd(X)∣.
The following values are dataset-equal averages on the 56-view common panel at 50 epochs; Time-RCD is zero-shot.
For CATCH, whose evaluation is stochastic, the same measure is 0.043 on the archived scoring pass. DCdetector is excluded for the readout issue described below.
Your DCdetector example prompted an implementation audit. Our submitted adapter averages a temporal softmax over the same axis, making its readout constant in exact arithmetic. The approximately 1.9-point mean absolute matched-AUC change comes from ranking floating-point differences, not meaningful score variation.
We will remove this implementation from behavioral comparisons and replace the USAD–DCdetector interpretation in Section 6. After exclusion, the main result is 19/48 rather than 19/54 model–dataset cells, with the maximum matched effect unchanged at 14.37 points. The model-level correlation between response strength and mean absolute detection change is 0.9349.
We will report paired sensitivity alongside rank separation and consistently distinguish low CT from pointwise invariance.