Detailed review: In the current state, I would reject this paper. Although an interesting problem is investigated, the limited technical contribution and poor presentation make me believe that the paper is not yet ready. If the presentation is substantially improved during rebuttal, I would update my rating to a weak reject, since I feel like the technical contribution can not be changed substantially within a limited time frame. I do believe, however, that this is interesting work that could benefit from additional work and rigour.
Below, I first list the main questions I had while reading your paper, after which I give some small comments and suggestions that could be useful for a camera-ready version/next iteration.
Q1: Your test measures whether there is a precursor in the existing benchmarks. For an anomalous event Ae=[te,ue] and width W, would it be possible to set Pe(W)=[te,te+W) and the remaining anomaly is then Ae′=[te+W,ue] (where you deal with anomalous intervals that are shorter than W)?
Q2: The precursor test (section 3.2) in some sense boils down to computing a shift in features/sequence characteristics of two consecutive windows. How do you guarantee this test catches all the precursors, since it can also be interpreted as a model to detect precursors? Additionally, why did you not use any other model (change point detection, self-supervised model, ...) to verify whether the pre-onset window is a visible precursor?
Q3 [smaller question]: is it possible that some information is missing from table 2 and figure 4? Section 4.3 refers to these results for the discussion, but mentions things like "On SMD, the rejection rate halves within 10 minutes" and "On 3W, it stays above 0.91 at every level up to 200 seconds". However, these tables and figures do not show absolute rejection rates at different time intervals of the onset (and no results at all about 3W). I quickly scanned the appendix B2 but could also not find this information.
Q4: The related work section mentions that "Pointwise F1 poorly represents interval-valued anomalies, whereas point adjustment may substantially inflate performance by counting an entire segment as detected after a single hit" (line131). To deal with this, MPA is introduced to better measure whether the precursor is detected. I have several questions regarding this:
Q4a: Did you evaluate whether MPA is more reliable than other metrics? Now, you only show the results using MPA without actually evaluating it.
Q4b: Why did you not evaluate using existing anomaly detection metrics (such as range-based, affiliation-based, VUS, which were referenced)? You could also apply minor adjustments to these to not take into account late detections (e.g., for affiliation, you could make sure that the affiliation regions always and at an anomaly). In addition, the #todo Tatbul.
Q4c: to avoid MPA being a pointwise metric, you average the anomaly scores across an anomalous interval. Could you also do this for F1, in which case it could also be a reliable metric for precursor detection?
Q5: Can you show the precursors on real time series? Now you only show some trivial synthetic examples. It would be interesting to see some of the samples in which there is a precursor according to the test and some samples in which the test fails.
Minor comments/suggestions:
- The abstract and introduction (especially first paragraph) assume a precise definition of the anomaly onset and pre-onset and near-onset, without actually defining it. A short description of what the "onset" means would be helpful. You could, for example, also interpret it as the complete anomalous event. The te notation also does not necessarily help, because it is not clear that te is the start of the anomalous event.
- Throughout the text, it was several times mentioned that the detectors are causal without any further explanations. However, most of the anomaly detectors do not leverage causal relations. I suggest dropping causality from the text, to avoid this ambiguity and simplify the text.
- The research questions are very nicely posed and come back through the methodology. This unfortunately stops at the experimental evaluation (section 4), where you would expect the questions to be answered. I believe this part can be restructured. The main thing would be to formulate a clear and explicit answer to the research questions, beyond simply mentioning the numbers in the tables.
- Several non-trivial concepts (event-placebo exchangeability, paired window exchangeability) are mentioned in passing, followed by some theoretical claim. There is no additional explanation as to why the claim follows the concept, nor is there a reference for the concept. Adding this would substantially improve understanding.
- Line 213: a simple intuition about what the rejection rate is (i.e., the percentage of R and Q that are rejected from showing a difference) would help the understanding. I could also be useful to add an (inline) formula of this.
- In fig. 3: there are only 5 out of 8 morphologies tested, without argumentation or clear reference to section B1.1 where all morphologies are shown. It would be useful to already explicitly mention in the setup of section 4.2 that this information is available in appendix, but also an argumentation as to why the specific subset of morphologies is shown and not everything.
- Line 355: discuss why are only 500 out of 537 events eligible. It is not clear what the criteria for being eligible is. It is not having a precursor, since otherwise all 500 events would be in P.
Brief Update on Our Response to Reviewer Tery’s Comments
Thank you very much for your careful review and your positive comments on the paper’s readability, structure, and presentation of the research questions. We appreciate your encouragement and your questions about concepts and notation that would benefit from further clarification.
Regarding Eq. (1), Σi denotes the shrinkage-regularized covariance matrix of statistical descriptors estimated from normal-operation data, and Σi−1 is its inverse. Intuitively, this term helps us measure how far a window departs from normal operation while accounting for the typical variability of the descriptors and their correlations. We will explain the meaning and role of these symbols more explicitly next to the equation.
Regarding “placebo calibration,” the “placebos” are control windows drawn from normal-operation records of the same device. We select time points within these records as hypothetical anomaly onsets and apply exactly the same calculations used for windows preceding actual anomalies. These time points simply serve to align the analysis windows; the underlying data remain normal-operation records. This provides a reference for how much change can arise from normal variation alone. We then compare the changes observed before an actual anomaly against this reference to assess whether they exceed what is typically observed in matched normal data. This is the basic idea of placebo calibration.
In addition, motivated by the limitations identified in this paper regarding precursor evidence and its validation in existing benchmarks, we have been developing a new real-world multivariate time series dataset. Data preparation was still underway at submission and is now complete. The dataset comprises multivariate time series collected from 48 electromechanical devices operating under real-world conditions, together with the corresponding system fault information. We will introduce this dataset for anomaly prediction and detection in our forthcoming response, with the aim of providing additional real-world data for precursor analysis and anomaly prediction evaluation. We will present this dataset in the appendix as a supplementary resource; its inclusion will not alter the paper’s main experiments or conclusions.
We will revise the main text to explain these concepts, the notation, and the overall research approach more clearly, and provide you with a more complete response. We need a few more days to complete this process and wanted to share this progress update in the meantime. We will submit our detailed response as soon as possible, and in any case before the deadline. Thank you again for your careful review, encouragement, and patience.