- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a two-stage anomaly detection pipeline for operational KPI data. The first stage combines nine direction-free deviation features into one score and picks the threshold that keeps as few observations as possible while missing no labeled development event. The second stage applies a rule that encodes the harmful direction, a rolling or seasonal reference, a robust threshold, a persistence requirement, and gap closing. An LLM can translate operator policies into such rules. On real backbone-link KPI data with 66 evaluation events, VeraTS ranks first in 10 of 12 category-metric columns against 34 baselines.
Critical strengths
The central point is real and practically important: the same waveform can be an anomaly under one policy and benign under another. The method is deterministic, auditable, and cheap. The ablations of direction and persistence (Fig. 5) and the parameter-sensitivity study are informative, and the baseline set is broad.
Critical weaknesses
-
The comparison is not on equal terms. VeraTS receives a supplied policy that specifies the harmful direction, reference, persistence, and closing, and its first-stage threshold is selected on labeled development data. The baselines are generic detectors that never see the policy. A stronger controlled comparison would use each competitive baseline's scores as the first-stage candidate mask and then apply the same second-stage policy rule. That would separate the value of VeraTS's screening features from the value of the policy-conditioned diagnosis.
-
The value of the first stage is not shown. No result reports the second stage without the gate, so it is unclear whether screening helps accuracy. The gate keeps 32.8% of observations, and Sec. 3.3 states that the current implementation still computes reference features over the full trace, so the cost argument in Eq. 10 is not realized in the evaluated system.
-
The Event-F1* comparison is retrospective. For each method and category, the best percentile between 90 and 99 is chosen using evaluation labels (Sec. 4), and Fig. 4 marks VeraTS's "label-selected maxima". The F1 ablations in Fig. 5 are more permissive still, using unrestricted per-series maxima. This does not affect the VUS results, but it makes the four Event-F1* wins oracle operating points rather than deployable estimates. Event-F1* is also a point-adjusted metric, which is known to overestimate detection quality (Kim et al., AAAI 2022). Because width-48 closing can bridge predictions across long gaps, its interaction with the point-adjusted Event-F1* metric may further increase event-level scores.
-
The abstract and introduction present LLM translation of operator policies as a stated component and contribution of the method, but the experiments use "supplied rules". No result in the main text measures whether the LLM produces correct rules.
-
All real-data results come from a single dataset with 66 evaluation events, with no evaluation on a standard public benchmark. The main text does not say how many distinct incidents or services these events represent, and no uncertainty estimates are reported, so it is hard to judge how general the findings are.
Kim, S., Choi, K., Choi, H.-S., Lee, B., and Yoon, S. (2022). Towards a Rigorous Evaluation of Time-Series Anomaly Detection. In Proceedings of the AAAI Conference on Artificial Intelligence, 36(7), 7194–7201.
Questions for authors
- How do competitive baselines perform when their scores are used as the candidate mask for the same second-stage rule?
- What are the event-F1 results with thresholds fixed on development data rather than chosen on evaluation labels?
- How does the second stage perform without the first-stage gate, in accuracy and in runtime?
- How were the rule parameters selected, and on which data?
- How accurately does the LLM translate operator policies into rules, compared with a deterministic parser or hand-written rules?
- Can the screening stage be evaluated on a public anomaly detection benchmark?
Discussion Reminder
Dear Reviewers,
Thank you for your efforts so far. This is a friendly reminder to review the authors' rebuttal and respond to their comments as soon as possible, before the discussion deadline (November 18 AoE). Please also indicate whether your concerns have been addressed. Thank you for your time and cooperation!
Best regards,
AC