- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SCAR, a method for multivariate time-series anomaly detection that addresses the problem of normal behavior differing across operating regimes. The method first decomposes each time window into slowly varying state information and rapidly varying local information, then retrieves normal reference segments in two stages based on the operating regime and local context. Finally, reference distance, state novelty, and reconstruction errors at two scales are mapped onto a common scale and combined into an anomaly score. The paper compares SCAR with 24 baselines on five main datasets and uses the TEP dataset to examine the relationship between retrieved references and actual operating regimes. Overall, the method achieves competitive detection performance.
Critical strengths
-
The research problem is practically relevant, and the motivation is clear. When an industrial system switches operating regimes, the same local variation may change from normal to anomalous. Retrieving normal segments from similar operating regimes before making local comparisons is a reasonable and intuitive design.
-
The main detection results are competitive. In Table 1, SCAR achieves the highest AUROC on all five datasets, with average AUROC and AP values of 0.7760 and 0.3666, respectively. In Table 2, the full method also substantially outperforms the global retrieval variant, providing experimental support for combining operating regimes with local context.
-
The experiments cover a broad range of settings and acknowledge the method’s limitations. Beyond the main benchmarks, the paper includes experiments on 530 TSB-AD time series, an analysis of TEP with ground-truth operating-regime labels, and experiments on contamination, ablations, and computational cost. The authors also acknowledge that fusion and purification do not help in every setting, allowing readers to understand the method’s practical trade-offs.
Critical weaknesses
-
The final scoring method is inconsistent with the explanations accompanying the case-study figures. Equation (11) on page 7 and the description on page 18 state that the final score is a fixed, equally weighted average of four diagnostic scores. Page 17 also explicitly states that prediction errors are used only for auxiliary training and do not contribute to the final score. However, Figures 5–9 on pages 29–33 show time-varying “fusion weights” and explain anomaly scores using long- and short-term prediction branches. These internal inconsistencies in the reported method and results indicate that the manuscript is not ready for submission.
-
The claim of stable calibration across operating regimes lacks independent validation. Appendix F first uses the normal set A to determine quantile thresholds and then computes regime-specific false-positive rates on that same set A. This analysis can describe differences across operating regimes within those data, but it cannot establish that false-positive rates remain controlled on new normal data. Moreover, Table 3 reports a tail error of 0.0572 at the 95% threshold, meaning that the mean absolute deviation of regime-specific false-positive rates from the target rate of 5% is 5.72 percentage points. The standard deviation of false-positive rates across operating regimes is 7.31 percentage points. These values are large, and independent testing and appropriate comparisons are needed to support the claim of stable calibration.
-
Variability across experiments and execution configurations are insufficiently documented, making some discrepancies difficult to explain. The first three groups of sensitivity experiments in Figure 4 on page 27 are all labeled as reusing Stage-A, yet the results corresponding to the same default parameters differ: 0.805, 0.805, and 0.798 for MSL, and 0.558, 0.571, and 0.571 for SMAP. These differences exceed rounding precision. The paper does not adequately explain these discrepancies and does not report variability across repeated runs for the main results. In particular, the full method’s average AP exceeds that of kNN by only 0.0069, so the available results are insufficient to determine whether this advantage is stable.
Questions for authors
The weaknesses above raise concerns about the correctness of the reported results. The model architecture appears to be described inconsistently across the manuscript, and the authors have not released the code. Together, these issues raise doubts about the reliability of the results, including concerns that they may be inaccurate or fabricated. The authors should provide sufficient experimental evidence to address these weaknesses.
3 comments
Response to Reviewer jiB8 Part2
2. Lack of held-out normal-data validation for cross-regime calibration
Thank you for pointing out this issue. The original Appendix F determined thresholds and calculated false-positive rates on the same normal set, without examining threshold performance on new normal data. Your interpretation of Table 3 is also correct: a target deviation of 5.72 percentage points and a cross-regime standard deviation of 7.31 percentage points are insufficient to support the claim of "stable calibration across operating regimes."
To address this issue, we added a temporal holdout evaluation of normal data covering six TEP modes: a normal segment outside the training data determines the alarm threshold, and a separate, later normal segment that has not participated in training, calibration, or threshold setting is used to evaluate the false-positive rate. Below, we refer to these two segments as the "threshold-setting set" and the "held-out evaluation set," respectively. We compare two settings that use either the normal training set or the threshold-setting set as the reference for score calibration; both determine the final alarm threshold on the threshold-setting set and evaluate it on the same held-out evaluation set.
| Score calibration reference | Target false-positive rate | Overall false-positive rate on the held-out evaluation set | Standard deviation of false-positive rates across six modes | Mean absolute target deviation |
|---|---|---|---|---|
| Normal training set | 5% | 0% | 0 percentage points | 5 percentage points |
| Normal training set | 1% | 0% | 0 percentage points | 1 percentage point |
| Threshold-setting set | 5% | 0% | 0 percentage points | 5 percentage points |
| Threshold-setting set | 1% | 0% | 0 percentage points | 1 percentage point |
On the held-out normal time segments from the six modes in this evaluation, neither calibration-reference setting produced false alarms, and the observed false-positive rates were below the specified 5% and 1% levels. This adds an evaluation of the thresholds on new normal time segments; however, the target deviations remain 5 and 1 percentage points, indicating conservative thresholds and not establishing that the false-positive rates precisely match the targets or that either calibration setting is superior. We have accordingly corrected the wording "stable calibration across operating regimes," clarifying that this holdout evaluation supports the statement that "on the evaluated held-out normal time segments from the six modes, false-positive rates did not exceed the specified targets."
Manuscript changes (all marked in blue; page and line numbers refer to the revised manuscript compiled for this revision):
- p. 9, Section 4.4, lines 466-470; the calibration interpretations and note for Table 3, lines 480-485: distinguish the original same-set statistics from the added temporal holdout evaluation, retaining the original values and correcting their meaning.
- p. 23, Appendix F, lines 1240-1241; pp. 25-26, new Appendix F.1, lines 1326-1354 (including p. 25, Table 8, lines 1342-1349): describe the holdout evaluation procedure and report false-positive rates, cross-mode standard deviations, and target deviations.
- p. 27, Appendix G, lines 1409-1412; p. 26, Figure 3 caption, lines 1398-1399: correct the conclusions drawn from the original tail statistics and note that the figure uses the same normal set for threshold setting and evaluation.
Response to Reviewer jiB8 Part3
3. Differences at sensitivity-analysis default points and AUROC/AP variation across seeds
To address the consistency of default points and AP variation across runs, we summarize AUROC/AP for five seeds on each dataset and report the means and sample standard deviations. Metrics use a 0-1 scale.
| Seed | MSL (AUROC / AP) | PSM (AUROC / AP) | SMAP (AUROC / AP) | SMD (AUROC / AP) | SWaT (AUROC / AP) | Macro-average (AUROC / AP) |
|---|---|---|---|---|---|---|
| 7 | 0.804949 / 0.299655 | 0.806539 / 0.576169 | 0.593723 / 0.159295 | 0.824955 / 0.194226 | 0.859023 / 0.612654 | 0.777838 / 0.368400 |
| 17 | 0.783341 / 0.261014 | 0.803330 / 0.606482 | 0.642018 / 0.205683 | 0.818582 / 0.186698 | 0.871892 / 0.638867 | 0.783833 / 0.379749 |
| 42 | 0.790831 / 0.266845 | 0.812555 / 0.593396 | 0.561231 / 0.142509 | 0.819608 / 0.207402 | 0.861967 / 0.761772 | 0.769238 / 0.394385 |
| 2026 | 0.707529 / 0.206655 | 0.790244 / 0.574140 | 0.621037 / 0.165716 | 0.814948 / 0.192483 | 0.861082 / 0.754107 | 0.758968 / 0.378620 |
| 3407 | 0.791219 / 0.270258 | 0.803769 / 0.592517 | 0.594681 / 0.161450 | 0.818000 / 0.187936 | 0.844616 / 0.744978 | 0.770457 / 0.391428 |
| Dataset | AUROC mean ± standard deviation | AP mean ± standard deviation |
|---|---|---|
| MSL | 0.775574 ± 0.038830 | 0.260885 ± 0.033791 |
| PSM | 0.803287 ± 0.008167 | 0.588541 ± 0.013432 |
| SMAP | 0.602538 ± 0.030605 | 0.166931 ± 0.023388 |
| SMD | 0.819219 ± 0.003646 | 0.193749 ± 0.008242 |
| SWaT | 0.859716 ± 0.009794 | 0.702476 ± 0.070891 |
| Macro-average | 0.772067 ± 0.009406 | 0.382516 ± 0.010517 |
Sensitivity-analysis default points. With the same default hyperparameters, different random runs may still produce different scores. The maximum differences among the default points for MSL and SMAP are 0.6846 and 1.2855 percentage points, respectively; the corresponding AUROC standard deviations in the five-seed results above are 3.8830 and 3.0605 percentage points, respectively. Numerically, both default-point differences are smaller than the corresponding cross-seed standard deviations in the table.
Whether the AP advantage disappears with variation across runs. Your concern is whether the macro-average AP gain of only 0.69 percentage points in the original manuscript persists across runs. Taking the original kNN result of 35.97% as the reference, the macro-average APs of the five seeds above are 36.84%, 37.97%, 39.44%, 37.86%, and 39.14%, respectively. All exceed this reference, with gains ranging from 0.87 to 3.47 percentage points. The five-seed macro-average AP is 38.25% ± 1.05 percentage points, with an average gain of 2.28 percentage points. Thus, across these five sets of results in the table, the macro-average AP advantage over the original kNN result does not reverse.
4. Reliability of results and reproducibility materials
We submitted the code as supplementary material with the original submission. To make it easier for you to inspect the code, we have further organized it and provided the following anonymous access link to the GitHub repository, which can be browsed and downloaded without logging in:
Anonymous code repository: https://anonymous.4open.science/r/scar-review-code-0E9B/
The repository includes the model, retrieval and scoring implementations, configurations for the five main datasets, training and evaluation entry points, environment instructions, and calibration tests, supplementing the materials submitted with the paper.
We plan to release the model weights on Hugging Face in the future. At present, to comply with anonymous submission requirements and avoid inadvertently revealing author identities through hosting accounts or file metadata, we are not providing a weight-download link; we will add it after completing the anonymity checks. We apologize for any inconvenience and appreciate your understanding. The current anonymous code repository does not contain model weights.
Response to Reviewer jiB8 Part1
Thank you for recognizing the motivation, detection performance, and experimental coverage of our work, and for identifying inconsistencies between the method description, case interpretations, and experimental records. Your central concern is whether these elements correspond to the same reproducible system. Accordingly, we have checked the scoring implementation, case-visualization procedure, and saved results, and supplemented them with a holdout evaluation of normal data and repeated runs under a fixed protocol. Below, we explain the verified facts, supplementary results, and aspects that still require correction.
1. Inconsistencies between final scoring and the interpretation of case figures
1.1 The auxiliary prediction task and reconstruction components in the case studies
The inconsistency you identified arose from the component names and explanations in Appendix M. The two scale-specific components used during SCAR inference are masked-reconstruction errors at patch sizes 8 and 32, measuring the recovery error of observed segments after masking. Next-patch prediction serves as an auxiliary training task during representation learning; its error does not enter the final anomaly score.
We have redrawn the case figures for all five datasets, replacing the original "Short/Long prediction" labels with Masked reconstruction (p = 8) and Masked reconstruction (p = 32), and rewritten the case analyses around the temporal behavior of events, multiscale reconstruction responses, memory deviation, and state novelty. The same 15 case windows are retained. For example, PSM C1/C2 exhibit broad score responses with a prominent scale-32 reconstruction signal; the brief disturbance in C3 is accompanied by a local peak and an increase in the scale-8 reconstruction display share. The corresponding explanations now agree with the actual scoring path.
Manuscript changes
1.2 Fixed equal-weight fusion and time-varying display shares
Equation (11) uses fixed equal-weight fusion. Denoting the four calibrated diagnostic components by uj(t), the final anomaly score is:
S(t)=41∑j=14uj(t).
Each component's coefficient remains 1/4, while its value varies over time. The varying proportions in Appendix M come from post-hoc visualization. The original manuscript called them "dynamic fusion weights," without clearly distinguishing display proportions from the model's scoring coefficients. We have renamed the two panels Display share (post-hoc softmax) and Allocated display score, respectively, and added their computational definitions.
Specifically, each diagnostic component and the final score separately use points labeled as normal in the test set of the same run as the percentile reference. A moving average of width 5 produces the component display curves qj(t) and the final display curve g(t). A softmax with temperature 0.1 is applied to the component curves, followed by a moving average of width 7 and normalization at each time step, yielding the display shares aj(t) in row (d). The thickness of each colored band in row (c) is:
bj(t)=aj(t)g(t),∑jbj(t)=g(t).
Thus, the stacked panel shows a post-hoc allocation of the final display curve across diagnostic signals. The additive terms in the actual score are uj(t)/4, which have a different meaning from bj(t) in the figure. The percentile transformations and softmax above are used for plotting; the detector still calibrates against training references and uses the equal-weight scoring in Equation (11). The revised legends, panel names, and case explanations all use this consistent definition.
Manuscript changes