- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SAGE, a training-free team of prompted LLM agents for univariate time series anomaly detection. Four Analyzers (point, structural, seasonal, pattern) call numerical tools such as STL, CUSUM, FFT and SAX. A Detector combines their outputs, and a Supervisor writes a report. The Detector also sees synthetic contrast pairs, built by injecting nine rule-based anomaly types into normal training segments.
Critical strengths
- The paper is honest about many of its limits (5% test portions, unequal budgets, uncalibrated confidence).
- Sensible split of labour: tools compute, the LLM judges.
- The output (interval, type, explanation) is closer to what an operator needs than a bare score.
- Broad set of studies: ablations, backbones, zero-shot runs, a human study.
- Prompts are given in full.
Critical weaknesses
-
The test portion of KPI and WSD is only 5% of each series (the rest is dropped for API cost). For WSD that is about two 400-point windows per series. Many windows will have no anomaly and per-series F1 will be unstable. The main claim is therefore weakly supported. Please evaluate on the full test splits or at least a much larger fixed sample.
-
It is not stated how the scores of the eight ML/DL baselines were turned into binary labels for Point-F1 (oracle threshold, percentile, or something else). This choice decides all of Table 1 and needs to be documented.
-
The baselines look weak. Plain STL residuals (Table 8, 53.2 average Point-F1) beat every baseline in Table 1 (best is 46.08). Matrix Profile scores only 10.2. TAMA gets 3.82 on Yahoo but 65.69 on the synthetic set, which suggests a failed reproduction.
-
The gains come from the specific LLM more than from the design. Point-F1 goes from 66.26 (Sonnet 4.6) to 48.20 (GPT-5.1) and 10.57 (Qwen3.5-9B). With GPT-5.1, SAGE is barely above LSTM-VAE (46.08). No open-weight model comes close, so the results cannot be reproduced without a proprietary model, and no code is released.
-
The ablation is mixed. Specialization helps on Yahoo, but on KPI and WSD removing it raises three of four metrics (e.g. WSD PA-F1 98.87 vs 95.68), and the paper says it cannot fully describe that condition (Appendix E.3). Why would the specialized design hurt on two of three datasets?
-
The "diagnosis" claim is not backed by real data. Type diagnosis is tested on 156 synthetic cases made with the same injection rules that shaped the Analyzer prompts (somewhat circular), and on a single-author annotation of 300 intervals, of which 251 are point anomalies. There are no real labels for structural, seasonal or pattern types. Also, the "recall" in Table 2a only measures whether the Analyzer returned some overlapping interval, not final detection.
-
All decision logic sits in hand-written prompts (the cut at 50, the spike-cluster bonus, "typically less than 5% of points are anomalous"). It is not said how these were chosen or whether test series were looked at. The Detector may also report intervals that no Analyzer proposed, and nothing checks them against tool evidence. How often does this happen?
-
The human study has 10 raters and 25 samples, with ground truth shown next to the predictions, so "Accuracy" mostly re-measures detection. Only SAGE writes a structured report, so blinding is doubtful, and the Wilcoxon test treats 250 dependent pairs as independent.
-
The novelty is limited: an orchestration of standard tools around a hosted LLM, which the paper itself says is "not a new numerical detector". Multi-agent and retrieval-based LLM detectors (TSAD-Agents [1], LLMAD [2]) already exist.
-
The cost is high: about 80 s per 400-point window and about 27K tokens per window. The total cost of the main run, and of each LLM baseline, is not given.
References: [1] Xu, X., Wang, H., Liang, Y., Yu, P. S., Zhao, Y. and Shu, K. (2026). Can Multimodal LLMs Perform Time Series Anomaly Detection? (TSAD-Agents). The ACM Web Conference 2026 (WWW '26). arXiv:2502.17812.
[2] Liu, J., Zhang, C., Qian, J., Ma, M., Qin, S., Bansal, C., Lin, Q., Rajmohan, S. and Zhang, D. (2024). Large Language Models can Deliver Accurate and Interpretable Time Series Anomaly Detection (LLMAD). arXiv:2405.15370.
Questions for authors
-
How were baseline scores thresholded in Table 1?
-
How many test windows per dataset contain at least one anomaly? What are the results on the full test splits?
-
Which runs produced Table 1 and which Table 13? Why do only Point-F1 values match?
-
How were the Detector rubric and the Global Rules chosen? Did anyone inspect test series?
-
How does SAGE perform with an open-weight backbone of reasonable size?
-
What is the total token and dollar cost of the full benchmark, for SAGE and for each LLM baseline?
2 comments
Thanks to the authors for the detailed reply. I could only check it against the submitted PDF, so I have not verified the changes promised for the revision.
-
Test scope: The new counts are useful, but they show that the KPI and WSD results are based on a handful of series. Were all the baselines evaluated on exactly the same series? It would also help to see some spread across series, since one series can move the average quite a bit.
-
Thresholds: The answer to Q1 changes how Table 1 should be read. If only SAGE gets a cutoff tuned on the test labels while the LLM baselines do not, the comparison with them is not on equal terms. Please add SAGE with a fixed cutoff next to Table 1 and state clearly in the paper how the cutoff is chosen for each method.
-
TAMA and STL: The explanation for the low TAMA score is plausible, and I withdraw my remark about a failed reproduction. STL is a strong baseline and should appear in Table 1 under the same protocol as the others.
-
Backbones: I missed that Table 12 already has Qwen3.5-27B, so Q5 is answered. Still, the gains seem to depend on the hosted model, and the benefit of specialization is clear only on Yahoo. The claims should be worded accordingly.
-
Diagnosis and human study: The reply agrees with my reading but adds no new evidence. The abstract should say only what was measured (a combined rating on a small number of cases), and diagnosis on real non-point anomalies is still untested.
Overall, the rebuttal improves the documentation but does not add new experiments. The test-label cutoff for SAGE and the small number of series behind the KPI and WSD results still weaken the main claim, so I keep my rating at weak reject. A fixed-cutoff comparison on the same series, with some spread, would change my view.
Response to Reviewer hqsh
Thank you for the careful review. We address the concerns in order.
1. Test scope (W1, Q2).
We used the final 5% of KPI/WSD for testing following LLMAD's setting (Sec. 4.1) and to limit API time and cost on long series. SAGE uses the first 70% for normal references and excludes the middle 25%; splits and file selection differ. The Sonnet runs cover 19 KPI and 18 WSD series, with 318 and 52 non-overlapping 400-point windows; 54 and 20 contain anomalies. Series with no labeled anomaly in the final 5% were excluded. We will disclose these rules and counts and limit conclusions to the evaluated portions. We do not have full-partition results.
2. Thresholds (W2, Q1).
SAGE uses training data for normal references but no separate anomaly-labeled validation set for cutoff selection. The Detector scores selected intervals, not every time point, so point-score calibration procedures do not transfer directly. For offline evaluation, SAGE selects a cutoff per test series on a 0.01 confidence grid to maximize raw Point-F1 using test labels. The ML/DL evaluations maximize the same objective over 100 thresholds between the 50th and 99.9th percentiles of each series' test scores. One binary prediction supplies all four metrics; only Point-F1 is optimized. This is a post-hoc oracle comparison, not deployment performance. Users can use types and evidence to set a domain-specific cutoff; its performance is untested. The grids differ, and direct-output LLM/VLM methods retain method-specific post-processing. We will document each rule.
3. Baselines and TAMA (W3).
STL residuals reach 53.2 mean Point-F1, above every ML/DL row in Table 1 (best 46.08). SAGE is about 66.3 in that comparison, which does not isolate the LLM's contribution or establish superiority over every lightweight method. On 355 Yahoo series, TAMA's saved outputs have 2.23% mean point precision and 63.80% recall: many false positives explain its low Point-F1. The synthetic task differs from Yahoo. We will describe the evaluated TAMA adaptation without generalizing this result to the method as a whole.
4. Backbones, ablation, and contribution (W4, W5, W9, Q5).
Mean Point-F1 is 66.26 with Sonnet 4.6, 48.20 with GPT-5.1, 42.84 with Qwen3.5-27B, and 10.57 with Qwen3.5-9B. The best result relies on a proprietary backbone. Removing specialization changes Point-F1 from 79.69 to 41.08 on Yahoo, 60.97 to 59.22 on KPI, and 58.12 to 55.54 on WSD, but raises the other three metrics on KPI/WSD. We will discuss this mixed result. Our contribution is the procedure for aggregating family-specific numerical/visual evidence, normal-derived contrasts, and support/conflict-aware diagnostic records. We do not claim novel tools or model architecture; the ablation does not isolate compute or backbone effects.
5. Diagnosis and human study (W6, W8).
The 156 injected cases share the Analyzers' type definitions, although synthetic ICL was disabled. Table 2's “recall” measures overlap with the relevant Analyzer evidence interval, not final Detector recall or independent accuracy on natural anomaly types. The 300-interval annotation is single-author and point-dominated; we will clarify aggregation and scope. Ten raters assessed 25 cases with method names hidden and order randomized, but reference and predicted intervals visible. Ratings reflect combined detection/report quality, not verification of every explanation. The Wilcoxon test treats repeated rater/case observations as independent; we will emphasize descriptive rater-level results.
6. Detector rules (W7, Q4).
The prompt's 50-point minimum evidence score determines what the Detector reports and differs from the evaluation cutoff. Spike-cluster guidance addresses dilution of individual z-scores in rolling statistics; “typically <5%” is a sparsity prior. The authors did not inspect benchmark test series or labels when choosing these rules, and we do not claim the constants are optimal. The Detector may report a visually supported interval outside Analyzer candidates. We will clarify this behavior; its frequency has not been measured, so we will not say every output was proposed by an Analyzer.
7. Tables and cost (W10, Q3, Q6).
Table 13 retained full-metric backbone values from an earlier analysis. Its Sonnet Point-F1 matches Table 1 after rounding, but some PA-F1, Affiliation-F1, and Delayed-F1 entries differ. Table 1 is our current main comparison; we will remove Table 13 and retain the Point-F1 backbone comparison. Recorded Sonnet latency is about 80 s per 400-point Yahoo window. One Yahoo A1 run used about 22.4K input and 4.9K output tokens per window; these are not whole-study averages or total costs. Total costs require usage logs. We make no cost-efficiency claim.