- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper defines a new task for time-series anomaly detection, DPE (deployment-time performance estimation): estimating the instance-level performance of an anomaly detector on unlabeled deployment instances. The proposed method, TimeDPE, combines a 29-dimensional hand-crafted surrogate representation—comprising 17 dimensions for score distributions, 5 for temporal structure, and 7 for the relationship between scores and the original sequence—with GP-RQ regression. On TSB-AD-U, which contains 170 source–detector pairs and 8,600 exact leave-one-out predictions, it reduces MAE by 40.2% relative to the HMT baseline. It also achieves a 61.4% reduction in an SDR radio EMD-monitoring use case.
Critical strengths
-
The task definition is clear and valuable: it is explicitly distinguished from unsupervised model selection, which only ranks models rather than estimating absolute performance. The Over metric (the proportion of overestimation) also reflects a proper understanding of deployment risks.
-
The evaluation protocol is rigorously designed and the analysis experiments are convincing: exact LOIO with no sampling noise; 20,000 bootstrap confidence intervals; cross-validation across four target metrics. Appendix Table 2 shows that, with the representation fixed, replacing the estimator (different GP kernels, KRR, or Ridge) barely changes the results, indicating that the main contribution comes from the surrogate representation. The Spearman correlation (\rho=0.98) between sufficiency (S_\tau) and performance gains quantitatively explains “why it works.” The failure analysis—19/170 cases with at most 20 instances, and correlations of 0.583 versus 0.806—is honest and internally consistent. Furthermore, a controllable real-world experiment is well designed and verifies the effectiveness of the proposed method.
-
Reproducibility is excellent: all 29 signals are fully defined, an anonymized code repository is provided, and the hardware-pipeline details are complete.
Critical weaknesses
-
Although a new task is defined, the technical contribution is weak.
-
Lack of generalizability evaluation on distribution shift cases. The paper treats each TSB-AD data source as a “deployment setting” and uses variation among instances within a source as a proxy for deployment variation. However, instances from the same data source typically originate from the same data-collection process and are relatively homogeneous in distribution. In real-world deployment, the core challenge is precisely the distribution shift between historical evaluation data and the deployment target, such as new devices, new operating conditions, and sensor aging.
-
There is no analysis of the computational cost. Although instance-level performance estimation is valuable for real-world deployment, the authors note that performance can vary over time. A clear evaluation of the computational overhead—including feature-construction and inference latency—would help readers assess the method’s feasibility for online or near-real-time applications.
Questions for authors
-
Could the authors provide a DPE evaluation across different data sources (or under substantial distribution shifts)? If performance drops significantly, how should the applicability boundaries of the method be characterized? (W2)
-
What is the computational cost of each signal family? Does the marginal gain from (\Phi_{\mathrm{rel}}) over (\Phi_{\mathrm{dist}}+\Phi_{\mathrm{temp}})—a 3.8-percentage-point relative improvement in MAE—justify its additional complexity? (W3)
2 comments
Response to W3 & Q2 — Computational Efficiency
We thank the reviewer for this constructive suggestion. We have conducted a theoretical complexity analysis of all 29 surrogate signals, together with empirical inference latency measurements. We then discuss the accuracy–efficiency trade-off of Φrel.
Computational complexity.
| Signal family | O(n) | O(n log n) | Overall |
|---|---|---|---|
| Distributional (17D) | 14 | 3 | O(n log n) |
| Temporal (5D) | 4 | 1 | O(n log n) |
| Relational (7D) | 3 | 4 | O(n log n) |
| Full (29D) | 21 | 8 | O(n log n) |
The table above summarizes the feature-construction complexity of the full surrogate representation, which is O(n log n) overall.
For Φrel, the four window-based relational signals (A04–A07) may introduce substantial additional computational overhead in practice due to repeated window-level statistical computations across multiple scales, despite retaining asymptotic complexity.
Deployment-time inference latency.
We further measure TimeDPE's total inference latency. To isolate the additional cost of the four window-based relational signals (A04–A07), we compare three configurations: 22D (Dist + Temp, excluding all relational signals), 25D (adding the three linear-time relational signals A01–A03), and 29D (the full representation).
We measure the complete inference pipeline, including feature construction, preprocessing, and GP-RQ prediction, on 152 length-stratified time series from TSB-AD-U using an Apple M5 CPU.
| Representation | Median (ms) | P95 (ms) |
|---|---|---|
| 22D | 4.91 | 64.81 |
| 25D | 6.27 | 84.89 |
| 29D | 34.99 | 517.20 |
The median reflects typical inference latency, while P95 captures the slower tail of the latency distribution. The evaluated time series range from 1,000 to 900,000 time steps (median length ≈18,000), with longer sequences contributing to higher inference latency.
Accuracy–efficiency trade-off. The additional cost of the four window-based relational signals is substantial, while their aggregate benefit is relatively modest (38.6% vs. 40.2% MAE reduction over HMT for 25D and 29D, respectively). However, as highlighted by our surrogate complementarity analysis in Figure 5, the value of individual signals depends strongly on the source–detector pair. For example, extending 25D to 29D reduces MAE by 44.9% on Exathlon–MOMENT and 43.2% on Exathlon–MatrixProfile relative to TimeDPE with the 25D representation.
These findings suggest that whether to include computationally expensive surrogate signals depends on the specific deployment setting. In some source–detector pairs, these signals provide substantial complementary information that cannot be adequately captured by lower-cost alternatives. This also highlights a key insight from DPE for TSAD: the importance of considering deployment-specific characteristics rather than relying solely on aggregate performance. As noted in our Conclusion (Line 484), designing domain-specific surrogate representations is therefore an important direction for future work.
The complexity analysis of feature construction and inference latency has been added to Appendix A.4 of the updated manuscript.
We sincerely thank the reviewer for the careful evaluation of our work and the constructive feedback. The comments have helped us clarify our methodological contributions, better characterize the applicability boundaries of TimeDPE, and strengthen our empirical analysis. We provide our point-by-point responses below.
Response to W1 — Technical Contribution
Our goal is to establish a clear task formulation for DPE in TSAD and provide a simple, interpretable solution framework built upon established methods. Beyond the task formulation, our main technical contributions lie in the design of surrogate representations and the systematic analysis of their effectiveness and limitations. Specifically, we integrate three complementary signal families—distributional, temporal, and raw–score relational—based on our understanding of which observable patterns in TSAD may be informative of detector performance. We believe our method provides a strong, interpretable baseline that helps clarify which factors drive DPE performance in TSAD and offers a solid reference point for future work.
Response to W2 — Distribution Shift and Problem Setting
We would like to clarify that time-series instances from the same data source are not necessarily homogeneous. Even within a shared data-collection setting, variations in operating conditions, temporal patterns, and anomaly characteristics can lead to substantial differences in detector performance. As illustrated in Figure 1, instance-level VUS-PR can range from nearly 0 to 1 within a single source–detector pair. Such variability poses challenges to deployment reliability and is precisely what motivates DPE for TSAD.
Regarding generalizability within our defined problem setting, TimeDPE achieves positive gains over HMT across all 17 data sources (Appendix Table 4), demonstrating its effectiveness across diverse sources. While generalization under substantial distribution shifts (cross-source) is beyond the scope of this paper, we agree that evaluating it is valuable for clarifying TimeDPE's applicability boundaries. We provide additional cross-source generalization experiments and discussion in our response to Q1, as follows.
Response to Q1 — Cross-source Evaluation and Applicability Boundaries
Additional experiments. In contrast to the within-source leave-one-instance-out setting used in our main experiments, we conducted a leave-one-data-source-out evaluation across 17 sources and 10 detectors. For each detector, TimeDPE is trained on historical evaluation records from 16 sources and evaluated on the held-out source. HMT uses the same historical records for a fair comparison.
Results and applicability boundaries. TimeDPE reduces Macro MAE from 0.3163 (HMT) to 0.2612, achieving a 17.4% improvement. Notably, TimeDPE still achieves positive gains on 13 of the 17 held-out sources. However, cross-source gains vary substantially across domains, ranging from +93.9% on MGAB to −55.4% on Stock. On Stock, TimeDPE achieves a 62.7% improvement within-source but underperforms HMT by 55.4% under cross-source evaluation.
These findings suggest that the representativeness of historical evaluation records is important for reliable performance estimation. Cross-source transfer is therefore not necessarily infeasible, but TimeDPE's applicability may depend on how well historical records cover the target deployment instances. Exploring how out-of-distribution (OOD) detection methods could help assess this applicability boundary is an interesting direction for future work.