- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FDAS, a forecasting-based anomaly detection approach that uses disagreement among forecasts for the same target time obtained from different forecast origins. Combined with smoothing and normalization, the proposed scoring method improves anomaly detection performance over conventional forecast-based scoring in the reported experiments.
Critical strengths
- Using forecasts from different origins: Comparing forecasts for the same target time from different origins is an interesting idea. Using frozen TSFMs, the paper shows that a post-hoc scoring method can improve anomaly detection performance.
- Potential of forecasting-based TSFMs for anomaly detection: The paper demonstrates that forecasting-based TSFMs can be used for anomaly detection by changing only the scoring method. This may broaden the applicability of existing forecasting-based TSFMs.
Critical weaknesses
- Contribution of forecast disagreement: Table 3 shows that EWMA smoothing accounts for much of the performance improvement over conventional scoring, particularly on the univariate benchmark. Adding forecast disagreement provides only modest additional gains, which makes the contribution of disagreement itself less convincing.
- Practical advantage over AD TSFMs: Although the proposed approach demonstrates the potential of forecasting-based TSFMs for anomaly detection, the referee is not convinced that they offer a practical advantage over TSFMs specifically designed for anomaly detection.
Questions for authors
- Beyond the aggregate results in Table 3 and the significance tests in Appendix E.1, how broadly are the additional gains from disagreement distributed across the evaluation series? How often does adding disagreement improve or degrade performance relative to smoothed and normalized forecast-error scoring? Are the improvements concentrated in particular types of anomalies?
- Under what conditions should a forecasting-based TSFM with FDAS be preferred to an anomaly-detection TSFM? Please distinguish between reusing an existing forecaster and deploying one specifically for anomaly detection, and clarify the trade-off between detection performance and computational cost in these settings.
3 comments
Response to Reviewer TvjS (2/3)
This comment continues our response (2/3).
TvjS-Q1 (Distribution of the gains from disagreement)
Question 1: "Beyond the aggregate results in Table 3 and the significance tests in Appendix E.1, how broadly are the additional gains from disagreement distributed across the evaluation series? How often does adding disagreement improve or degrade performance relative to smoothed and normalized forecast-error scoring? Are the improvements concentrated in particular types of anomalies?"
In all 13 configurations, the gain from the disagreement track is positive on average for series that contain only sequence anomalies, which make up 86% of U and 93% of M series, and for the half of the series with the longest average anomaly lengths. We re-aggregated the stored per-series results as a post-hoc analysis, using the baseline in your question, the smoothed and normalised forecasting-error track (Ẽ1). The values below are the change in VUS-PR when the disagreement track is combined with Ẽ1 (FDAS − Ẽ1), averaged across configurations.
Share of series improved or worsened (Table 4, "All series" rows). Compare the bold Improved and Worsened columns. Ties arise mainly in series scored through the fallback, where FDAS and Ẽ1 coincide (Appendix C.3). Percentages may not sum to 100 because of rounding.
| Improved | Tied | Worsened | |
|---|---|---|---|
| U (8 configurations) | 37% | 25% | 39% |
| M (5 configurations) | 55% | 10% | 35% |
By anomaly type (point and sequence anomalies as labelled in TSB-AD; Table 4). Bold values show the mean VUS-PR change; parentheses give the number of configurations with a positive group mean (Appendix E.4).
| Series (U / M) | U: FDAS − Ẽ1 | M: FDAS − Ẽ1 | |
|---|---|---|---|
| Only sequence anomalies | 301 / 167 | +0.026 (8/8) | +0.041 (5/5) |
| Containing point anomalies | 49 / 13 | −0.005 (0/8) | −0.050 (0/5) |
By quartile of average anomaly length, Q1 the shortest and Q4 the longest (Table 21, Figure 17). Each cell gives the number of series and the mean VUS-PR change, FDAS − Ẽ1, in bold.
| Quartile | Q1 | Q2 | Q3 | Q4 |
|---|---|---|---|---|
| U | 92 / −0.015 | 83 / +0.006 | 88 / +0.063 | 87 / +0.035 |
| M | 45 / −0.050 | 45 / +0.102 | 45 / +0.053 | 45 / +0.031 |
By type, the group mean is positive in every configuration for series that contain only sequence anomalies (U 8/8, M 5/5) and in no configuration for series that contain point anomalies. Length shows the same pattern: in all 13 configurations, the group mean is negative in Q1 and positive in Q3 and Q4, which contain half of the series on each benchmark. In Q2, it is positive in five of the eight U configurations and in all five M configurations. Anomalies are shorter on U, with a median average anomaly length of 116 time steps against 241 on M, which corresponds to the smaller overall gain on U.
Separately, comparing the two single tracks, the difference VUS-PR(D̃) − VUS-PR(Ẽ1) is positively correlated with the average anomaly length in all 13 configurations (Spearman; Figure 16).
Revision: Section 5.2 (Table 4) summarises the distribution, and Appendix E.4 (Table 21, Figures 16–17) gives the full results by anomaly type and length.
Response to Reviewer TvjS (3/3)
This comment continues our response (3/3).
TvjS-W2 + TvjS-Q2 (Practical advantage over AD TSFMs; conditions and cost)
Weakness 2: "Practical advantage over AD TSFMs: Although the proposed approach demonstrates the potential of forecasting-based TSFMs for anomaly detection, the referee is not convinced that they offer a practical advantage over TSFMs specifically designed for anomaly detection."
Question 2: "Under what conditions should a forecasting-based TSFM with FDAS be preferred to an anomaly-detection TSFM? Please distinguish between reusing an existing forecaster and deploying one specifically for anomaly detection, and clarify the trade-off between detection performance and computational cost in these settings."
We answer the two points together because they are connected. Which of FDAS and an AD TSFM is more suitable depends on the deployment setting:
- If a forecaster already produces, at every time step, the H-step forecasts that FDAS needs, FDAS provides anomaly scores from those forecasts with no additional forward pass.
- If a model is deployed on univariate data solely for anomaly detection, TimeRCD has higher mean VUS-PR and is faster.
- On multivariate data, three FDAS configurations have higher mean VUS-PR than every AD TSFM compared in Table 2, and they processed all series of the evaluation split on one 32 GB GPU (RTX 5090).
- With their released implementations and settings, TimeRCD and TSPulse exceeded the memory of the same GPU on some multivariate series.
The times in 1) and 3) below were measured over the full evaluation split (U: 350, M: 180 series), with forecasting and AD TSFMs on an RTX 5090 GPU and FDAS scoring on one CPU thread.
1) Reusing an existing forecaster. If the forecaster already produces the H-step forecasts that FDAS needs, FDAS uses them as they are, and the only added cost is scoring: 10.5–21.5 s on U and 3.3–6.6 min on M over the full evaluation split (Appendix A.2, Table 5). For the same series, the scoring cost grows linearly with H.
2) An existing forecaster that outputs a shorter horizon. For the nine backbones whose output length can be set, increasing the output length from H = 1 to the selected H changed the forecasting time by a factor of 0.98–1.14, measured on the same ten evaluation series per benchmark (Table 6).
3) Deploying a forecaster for anomaly detection. Forecasting dominates the cost, and the total time ranges from 1.2 min to 31.2 h on U and from 13.3 min to 35.0 h on M depending on the backbone. On U, TimeRCD (0.95 min) has higher VUS-PR and is faster than every FDAS configuration. On M, of the three configurations with higher mean VUS-PR than the AD TSFMs, the two with the light TTM-r2.1 backbone take 13.3 min and 17.1 min in total (Table 5). CHARM's TSB-AD implementation obtains embeddings from a hosted API, so its time could not be measured on the same hardware and is not in the table.
4) Trade-off between performance and cost. The number of forecasts for a time step equals H, so we recomputed FDAS with the first H steps of the stored forecasts, from H = 2 up to the value selected on the tuning split. FDAS − Ẽ1 is 0.003–0.014 on U and 0.000–0.005 on M at H = 2, and 0.017–0.028 and 0.025–0.045 at the selected H, larger in all 13 configurations, although it does not increase at every step of H (Appendix E.4, Table 22, Figure 18). When a forecaster is reused, the backbone is fixed and the added cost is scoring, which grows linearly with H. In a new deployment, forecasting accounts for most of the total time; across the nine evaluated backbones, a larger H changed the forecasting time by a factor of only 0.98–1.14, whereas the total time in Table 5 varied widely across backbones. A new deployment should therefore weigh the total time of each backbone together with its detection performance.
Revision: the contributions in Section 1 and Section 5.2 state the reuse cost and the comparison for a new deployment; the measurements and conditions for 1)–3) are in Appendix A.2 (Tables 5–6), and the results for 4) are in Appendix E.4 (Table 22, Figure 18).
We hope these answers address your concerns. If anything remains unclear, we would be glad to clarify.
Response to Reviewer TvjS (1/3)
Thank you for finding the idea of comparing forecasts made at different origins for the same time step interesting, and for noting that changing only the scoring broadens the use of existing forecasting TSFMs. Your careful review and specific questions helped us sharpen the paper. Revised text is marked in blue in the revised PDF, and section, table and figure numbers below refer to it. We answer each point below.
TvjS-W1 (Contribution of forecast disagreement)
Disagreement improves mean VUS-PR beyond the smoothed and normalised one-step forecasting-error track (Ẽ1) in all 13 configurations, by 0.017–0.028 on TSB-AD-U (U) and 0.025–0.045 on TSB-AD-M (M), significantly in eight under unadjusted paired Wilcoxon tests (Appendix E.1). Smoothing and normalisation are components of the scoring layer together with the disagreement track, and the gain over conventional scoring (CS) comes from the layer as a whole. The submission did not report each component's contribution as a number; the revised abstract and main text now do.
Because the gains of smoothing and disagreement are not additive, the increment depends on whether smoothing is applied first. Without smoothing and with the same per-channel normalisation and channel maximum, the disagreement track alone outperforms the one-step forecasting-error track alone in 12 of 13 configurations (7/8 U, 5/5 M), and adding it raises mean VUS-PR by 0.07–0.10 on U and 0.03–0.07 on M (Table 3). Smoothing raises the one-step error track much more than the disagreement track (on U, +0.13–0.18 against −0.01 to +0.02), which is why the increment after smoothing is smaller.
Q1 below gives how this gain is distributed across series and anomaly types.
Revision: the abstract states the gain over conventional scoring with the same smoothing; the contributions in Section 1 and Section 5.2 state the disagreement increment with and without smoothing.