acceptodds
Under review as a conference paper at ICLR 2027

Deployment Performance Estimation for Time-Series Anomaly Detection

Abstract

Time-series anomaly detectors can exhibit substantial instance-to-instance performance variation even within the same data source. Yet conventional benchmark evaluations summarize performance over labeled collections and provide little indication of how a given detector will perform on a particular deployment instance. We define deployment-time performance estimation (DPE) for time-series anomaly detection as the task of estimating a detector's instance-level performance on an unlabeled deployment instance, given its historical evaluation records. To address this challenge, we introduce TimeDPE, which constructs a compact, label-free surrogate representation from the detector's anomaly-score sequence and the raw time series to capture performance-relevant information, and learns its relationship with detector performance from historical evaluations. Across 17 data sources and 10 anomaly detectors, TimeDPE reduces the macro-averaged mean absolute error of instance-level VUS-PR estimation by 40.2% relative to the conventional practice of predicting each target with the mean performance from historical benchmark evaluations. Moreover, we show that the surrogate representation plays a key role in TimeDPE's sample efficiency, enabling accurate performance estimation from limited historical evaluations.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series Anomaly Detection, Performance Prediction

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_fpx1OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper defines a new task for time-series anomaly detection, DPE (deployment-time performance estimation): estimating the instance-level performance of an anomaly detector on unlabeled deployment instances. The proposed method, TimeDPE, combines a 29-dimensional hand-crafted surrogate representation—comprising 17 dimensions for score distributions, 5 for temporal structure, and 7 for the relationship between scores and the original sequence—with GP-RQ regression. On TSB-AD-U, which contains 170 source–detector pairs and 8,600 exact leave-one-out predictions, it reduces MAE by 40.2% relative to the HMT baseline. It also achieves a 61.4% reduction in an SDR radio EMD-monitoring use case.

Critical strengths
  1. The task definition is clear and valuable: it is explicitly distinguished from unsupervised model selection, which only ranks models rather than estimating absolute performance. The Over metric (the proportion of overestimation) also reflects a proper understanding of deployment risks.

  2. The evaluation protocol is rigorously designed and the analysis experiments are convincing: exact LOIO with no sampling noise; 20,000 bootstrap confidence intervals; cross-validation across four target metrics. Appendix Table 2 shows that, with the representation fixed, replacing the estimator (different GP kernels, KRR, or Ridge) barely changes the results, indicating that the main contribution comes from the surrogate representation. The Spearman correlation (\rho=0.98) between sufficiency (S_\tau) and performance gains quantitatively explains “why it works.” The failure analysis—19/170 cases with at most 20 instances, and correlations of 0.583 versus 0.806—is honest and internally consistent. Furthermore, a controllable real-world experiment is well designed and verifies the effectiveness of the proposed method.

  3. Reproducibility is excellent: all 29 signals are fully defined, an anonymized code repository is provided, and the hardware-pipeline details are complete.

Critical weaknesses
  1. Although a new task is defined, the technical contribution is weak.

  2. Lack of generalizability evaluation on distribution shift cases. The paper treats each TSB-AD data source as a “deployment setting” and uses variation among instances within a source as a proxy for deployment variation. However, instances from the same data source typically originate from the same data-collection process and are relatively homogeneous in distribution. In real-world deployment, the core challenge is precisely the distribution shift between historical evaluation data and the deployment target, such as new devices, new operating conditions, and sensor aging.

  3. There is no analysis of the computational cost. Although instance-level performance estimation is valuable for real-world deployment, the authors note that performance can vary over time. A clear evaluation of the computational overhead—including feature-construction and inference latency—would help readers assess the method’s feasibility for online or near-real-time applications.

Questions for authors
  1. Could the authors provide a DPE evaluation across different data sources (or under substantial distribution shifts)? If performance drops significantly, how should the applicability boundaries of the method be characterized? (W2)

  2. What is the computational cost of each signal family? Does the marginal gain from (\Phi_{\mathrm{rel}}) over (\Phi_{\mathrm{dist}}+\Phi_{\mathrm{temp}})—a 3.8-percentage-point relative improvement in MAE—justify its additional complexity? (W3)

2 comments

We sincerely thank the reviewer for the careful evaluation of our work and the constructive feedback. The comments have helped us clarify our methodological contributions, better characterize the applicability boundaries of TimeDPE, and strengthen our empirical analysis. We provide our point-by-point responses below.

Response to W1 — Technical Contribution

Our goal is to establish a clear task formulation for DPE in TSAD and provide a simple, interpretable solution framework built upon established methods. Beyond the task formulation, our main technical contributions lie in the design of surrogate representations and the systematic analysis of their effectiveness and limitations. Specifically, we integrate three complementary signal families—distributional, temporal, and raw–score relational—based on our understanding of which observable patterns in TSAD may be informative of detector performance. We believe our method provides a strong, interpretable baseline that helps clarify which factors drive DPE performance in TSAD and offers a solid reference point for future work.

Response to W2 — Distribution Shift and Problem Setting

We would like to clarify that time-series instances from the same data source are not necessarily homogeneous. Even within a shared data-collection setting, variations in operating conditions, temporal patterns, and anomaly characteristics can lead to substantial differences in detector performance. As illustrated in Figure 1, instance-level VUS-PR can range from nearly 0 to 1 within a single source–detector pair. Such variability poses challenges to deployment reliability and is precisely what motivates DPE for TSAD.

Regarding generalizability within our defined problem setting, TimeDPE achieves positive gains over HMT across all 17 data sources (Appendix Table 4), demonstrating its effectiveness across diverse sources. While generalization under substantial distribution shifts (cross-source) is beyond the scope of this paper, we agree that evaluating it is valuable for clarifying TimeDPE's applicability boundaries. We provide additional cross-source generalization experiments and discussion in our response to Q1, as follows.

Response to Q1 — Cross-source Evaluation and Applicability Boundaries

Additional experiments. In contrast to the within-source leave-one-instance-out setting used in our main experiments, we conducted a leave-one-data-source-out evaluation across 17 sources and 10 detectors. For each detector, TimeDPE is trained on historical evaluation records from 16 sources and evaluated on the held-out source. HMT uses the same historical records for a fair comparison.

Results and applicability boundaries. TimeDPE reduces Macro MAE from 0.3163 (HMT) to 0.2612, achieving a 17.4% improvement. Notably, TimeDPE still achieves positive gains on 13 of the 17 held-out sources. However, cross-source gains vary substantially across domains, ranging from +93.9% on MGAB to −55.4% on Stock. On Stock, TimeDPE achieves a 62.7% improvement within-source but underperforms HMT by 55.4% under cross-source evaluation.

These findings suggest that the representativeness of historical evaluation records is important for reliable performance estimation. Cross-source transfer is therefore not necessarily infeasible, but TimeDPE's applicability may depend on how well historical records cover the target deployment instances. Exploring how out-of-distribution (OOD) detection methods could help assess this applicability boundary is an interesting direction for future work.

Response to W3 & Q2 — Computational Efficiency

We thank the reviewer for this constructive suggestion. We have conducted a theoretical complexity analysis of all 29 surrogate signals, together with empirical inference latency measurements. We then discuss the accuracy–efficiency trade-off of Φrel.

Computational complexity.

Signal familyO(n)O(n log n)Overall
Distributional (17D)143O(n log n)
Temporal (5D)41O(n log n)
Relational (7D)34O(n log n)
Full (29D)218O(n log n)

The table above summarizes the feature-construction complexity of the full surrogate representation, which is O(n log n) overall.

For Φrel, the four window-based relational signals (A04–A07) may introduce substantial additional computational overhead in practice due to repeated window-level statistical computations across multiple scales, despite retaining O(nlog⁡n)O(n\log n) asymptotic complexity.

Deployment-time inference latency.

We further measure TimeDPE's total inference latency. To isolate the additional cost of the four window-based relational signals (A04–A07), we compare three configurations: 22D (Dist + Temp, excluding all relational signals), 25D (adding the three linear-time relational signals A01–A03), and 29D (the full representation).

We measure the complete inference pipeline, including feature construction, preprocessing, and GP-RQ prediction, on 152 length-stratified time series from TSB-AD-U using an Apple M5 CPU.

RepresentationMedian (ms)P95 (ms)
22D4.9164.81
25D6.2784.89
29D34.99517.20

The median reflects typical inference latency, while P95 captures the slower tail of the latency distribution. The evaluated time series range from 1,000 to 900,000 time steps (median length ≈18,000), with longer sequences contributing to higher inference latency.

Accuracy–efficiency trade-off. The additional cost of the four window-based relational signals is substantial, while their aggregate benefit is relatively modest (38.6% vs. 40.2% MAE reduction over HMT for 25D and 29D, respectively). However, as highlighted by our surrogate complementarity analysis in Figure 5, the value of individual signals depends strongly on the source–detector pair. For example, extending 25D to 29D reduces MAE by 44.9% on Exathlon–MOMENT and 43.2% on Exathlon–MatrixProfile relative to TimeDPE with the 25D representation.

These findings suggest that whether to include computationally expensive surrogate signals depends on the specific deployment setting. In some source–detector pairs, these signals provide substantial complementary information that cannot be adequately captured by lower-cost alternatives. This also highlights a key insight from DPE for TSAD: the importance of considering deployment-specific characteristics rather than relying solely on aggregate performance. As noted in our Conclusion (Line 484), designing domain-specific surrogate representations is therefore an important direction for future work.

The complexity analysis of feature construction and inference latency has been added to Appendix A.4 of the updated manuscript.

Reviewer_uFcTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies deployment-time performance estimation (DPE) for time-series anomaly detection (TSAD). Given a frozen detector and historical evaluation records from the same deployment setting, the goal is to estimate the detector’s instance-level performance on an unlabeled target instance. Unlike historical mean transfer (HMT), which assigns the same estimate to every instance, DPE accounts for instance-level differences and estimates absolute performance rather than relative detector rankings.

TIMEDPE constructs a 29-dimensional label-free proxy representation from the raw time series and anomaly scores, comprising 17 score-distribution features, five temporal-structure features, and seven features describing relationships between the raw series and scores. GP-RQ then learns a mapping from these representations to performance using historical records.

Experiments cover 17 data sources and 10 detectors from TSB-AD-U, yielding 170 source–detector pairs and 8,600 predictions under exact leave-one-instance-out evaluation. Compared with HMT, TIMEDPE reduces macro-averaged MAE for VUS-PR from 0.1637 to 0.0979, a relative reduction of 40.2%; increases [email protected] from 40.3% to 71.9%; and reduces Over from 30.5% to 12.7%. Improvements are observed across all 10 detectors and all 17 data sources.

Critical strengths
  1. The problem is clearly defined and has practical deployment value. The paper explicitly distinguishes DPE from HMT and unsupervised model selection: HMT provides an aggregate estimate, model selection produces relative rankings, and DPE estimates a given detector’s absolute instance-level performance on an unlabeled target instance. This addresses a real gap in TSAD deployment, and the task definition has value in its own right.

  2. The experiments are extensive, the protocol is rigorous, and the results are consistent and reproducible. The claims are supported by 170 source–detector pairs, 8,600 predictions, exact leave-one-instance-out evaluation, strong baselines, ablations of the three feature families, sample-efficiency curves, failure analysis, and an EMD use case. The evaluation gives equal weight to each source–detector pair and reports bootstrap confidence intervals. Improvements are observed across all detectors and data sources, with substantial gains in [email protected] and Over. The code and appendix provide reasonably complete implementation details.

Critical weaknesses
  1. Methodological novelty is moderate; the main contributions lie in the task definition and empirical study. TIMEDPE combines handcrafted proxy features with GP-RQ, both of which are established tools. Its primary innovations are the formulation of DPE and the design of TSAD-specific proxy representations, rather than a new learning mechanism or theoretical contribution.

  2. Limited experimental coverage and restrictive assumptions weaken the claims of generality. The main experiments focus on TSB-AD-U and predominantly univariate time series. DPE relies on historical performance records derived from labeled instances under the same deployment setting, while cold-start conditions and labeling costs receive insufficient discussion. The 19 failures among 170 source–detector pairs lack systematic diagnosis. Direct comparisons with unsupervised model-selection methods under the same DPE protocol are also limited. These gaps call for more cautious claims about general-purpose deployment evaluation.

Questions for authors
  1. Could TIMEDPE be evaluated on TSB-AD-M or in a real multivariate deployment setting? How would the 29 proxy features extend to multiple channels, and would features describing relationships between channels be necessary?

  2. How does the method perform when very few historical instances are available, for example, N<5N < 5? Could it incorporate active sampling or strategies for acquiring a small number of labels? How could it be combined with unsupervised model-selection methods?

  3. Could the 19 failures among 170 source–detector pairs be characterized using diagnostic rules based on data properties, such as length, autocorrelation, and signal-to-noise ratio, or detector characteristics? At deployment time, how could one identify an inadequate proxy representation and fall back to HMT?

  4. Could TIMEDPE be compared with representative unsupervised model-selection methods in terms of absolute estimation error under the same DPE protocol? Such a comparison would help clarify the differences and complementarity between DPE and model selection.

  5. Although the full 29-dimensional representation with GP-RQ performs best, could the authors provide feature-importance and stability analyses to assess overfitting risks beyond the random-subset sufficiency analysis? Ridge also achieves a 32.4% improvement. Does this suggest that the proxy representation contributes more than GP-RQ?

  6. Although the results are consistent beyond VUS-PR, does the proxy-feature design depend specifically on range-aware metrics? Could the authors report confidence intervals for each data source and provide a more detailed failure analysis?

4 comments

We thank the reviewer for recognizing the value of the DPE problem formulation and the rigor of our empirical evaluation. Our point-by-point responses to the reviewer's concerns are provided below.

Response to W1 — Methodological Contribution

Indeed, we are not trying to introduce a new learning mechanism. Our goal is to establish DPE as a distinct problem and provide a clean, interpretable solution framework, supported by extensive empirical evaluations and analyses of its effectiveness and limitations. We believe this provides a useful foundation for further methodological developments in this emerging research direction.

Response to W2 — Generality and Deployment Assumptions

The concerns raised in W2 are further elaborated in Q1–Q6, and we address them individually in the corresponding responses below.

Response to Q1 — Extension to Multivariate Time Series

We agree that evaluating TimeDPE in multivariate time-series (MTS) settings is important. For our additional experiments on TSB-AD-M and the discussion of extending the 29 surrogate features to multivariate time series, please refer to our response to Reviewer M8wd (W1 & Q1) and the revised manuscript (see Table 6 in the Appendix)

Our results on MTS show that TimeDPE can achieve effective performance estimation using only the 22 anomaly-score-based surrogate features (23.0% Macro MAE reduction over HMT), without explicitly modeling cross-channel relationships. Nevertheless, we acknowledge that cross-channel relationships may provide additional performance-relevant information, and exploring their incorporation is a promising direction for future work.

Response to Q2 — Few-shot Performance, Label Acquisition, and Model Selection

We conducted additional few-shot experiments on three representative sources (LTDB, OPPORTUNITY, and UCR), using only N=2, 3, and 4 historical evaluation records with 20 random seeds. TimeDPE reduces Macro MAE over HMT by 6.24%, 11.53%, and 16.87% at N=2, 3, and 4, respectively. We also evaluated two previously excluded sources (SED and TAO), each containing only three instances, where TimeDPE achieves a 17.79% reduction over HMT under exact leave-one-instance-out evaluation. These results demonstrate TimeDPE's effectiveness even with very limited historical records in the evaluated settings.

These results hold under random sampling, without any selection strategy. Active sampling is complementary to TimeDPE and can be readily combined with it. More informative samples could further improve estimation quality in low-data regimes, but this is orthogonal to our contributions. Finally, combining unsupervised model selection with TimeDPE could potentially improve deployment reliability. However, their application settings are not directly aligned: unsupervised model selection methods operate without labels, whereas TimeDPE requires labeled historical records for training. Moreover, model selection itself could be a downstream application of TimeDPE. Exploring how these approaches could be combined effectively remains an open question.

Response to Q3 — Failure Characterization and Deployment-time Reliability.

We understand the reviewer's concern as asking whether potential failure cases of TimeDPE can be identified in advance to ensure reliable deployment, potentially through a fallback to HMT.

Our existing analyses provide some relevant insights. First, as discussed in our failure analysis (Lines 443–451 of the original manuscript), all 19 source–detector pairs where TimeDPE underperforms HMT contain at most 20 instances, suggesting that limited historical supervision is an important risk factor. Second, consistent with our surrogate sufficiency analysis (Lines 433–442), failed pairs exhibit weaker correlations between surrogate signals and detector performance recorded in historical evaluations than successful pairs (median maximum absolute Spearman correlation: 0.583 vs. 0.806). This suggests that assessing such correlations for a given source–detector pair could help identify potential risks before deploying TimeDPE. Additionally, Appendix Table 3 shows consistent improvements across all 10 detectors, suggesting that the failures are not systematically associated with any particular detector.

While our analyses provide some relevant insights, establishing systematic and reliable failure diagnostics remains a challenging problem in supervised learning (e.g., Gawlikowski et al., 2023). Identifying such cases at deployment time without ground-truth labels requires assessing the reliability of the performance estimator itself. Moreover, if certain observable data properties or detector characteristics could reliably identify failure cases, they could potentially be incorporated as additional surrogate signals to improve TimeDPE itself, rather than only serving as indicators for falling back to HMT. Developing such reliable diagnostics is therefore an interesting direction for future work.

Response to Q4 — Comparison with Unsupervised Model Selection

As discussed in the Related Work of our original manuscript, unsupervised model selection primarily aims to rank or select detectors, whereas DPE requires accurate absolute performance estimation. Even when performance regression is employed for model selection, its ultimate objective remains identifying the best-performing detector, which can rely on relatively coarse performance comparisons rather than precise absolute performance estimates. This distinction motivated our initial focus on HMT as a direct baseline for DPE, rather than adapting methods primarily designed for model selection.

Nevertheless, we agree that a direct empirical comparison under the DPE setting would help clarify the applicability of these approaches. We therefore adapted two representative regression-based methods, SATzilla (Xu et al., 2008; as implemented in TSB-AutoAD by Liu et al., 2025) and URegression (UReg) (Navarro et al., 2023), by directly using their internal performance predictions for comparison with TimeDPE.

We evaluated both methods using the same experimental protocol as TimeDPE, covering 17 data sources, 10 detectors, and 8,600 exact leave-one-instance-out predictions per method. The results are summarized below.

MethodMacro MAE ↓Gain vs. HMT ↑[email protected] ↑Over ↓
HMT0.1637—40.3%30.5%
SATzilla (adapted)0.127821.9%61.3%19.7%
UReg (adapted)0.127422.2%58.5%21.4%
TimeDPE0.097940.2%71.9%12.7%

Both adapted model-selection methods achieve meaningful absolute performance estimates and substantially outperform HMT. Nevertheless, TimeDPE achieves the lowest estimation error. Further details on the experimental settings and implementations are provided in the revised manuscript (Lines 322–342 and Table 1).

Response to Q5 — Feature Importance, Stability, and Overfitting

We agree that overfitting is an important concern under limited historical supervision. This concern was already considered in our design: we adopted handcrafted surrogate signals rather than end-to-end representation learning, and GP-RQ provides a smoothness prior suitable for low-data regression.

Our existing analyses provide relevant empirical evidence. First, the learning curves in Figure 4 show that TimeDPE maintains strong held-out estimation accuracy across different historical-data budgets, while requiring substantially fewer records than end-to-end alternatives. Second, our failure analysis shows that all 19 conditions where TimeDPE underperforms HMT contain at most 20 instances, indicating that limited historical supervision is an important risk factor. Moreover, Figure 5(a) shows that the most informative surrogate signals vary across source–detector conditions, motivating the use of complementary signals rather than relying on a small fixed subset.

To further investigate feature stability, we conducted an additional Permutation Feature Importance (PFI) analysis on UCR using 20 randomly sampled historical subsets at N = 5, 30, and 100. As N increases, the mean Spearman correlation between feature-importance rankings rises from 0.167 to 0.250 and 0.593, while held-out Macro MAE decreases from 0.1200 to 0.0869 and 0.0748. These findings suggest that limited supervision is associated with less stable feature reliance and higher generalization error. However, feature-importance instability should not be interpreted as direct evidence of overfitting, as it may also reflect training-sample sensitivity and correlations among surrogate signals.

Response to Q5 — Surrogate Representation vs. Estimator Head

Yes, this is consistent with our discussion in the original manuscript (Lines 417–421). Under limited historical supervision, the quality of the surrogate representation plays a more important role than the choice of estimator head.

Response to Q6 — Metric Generality, Confidence Intervals, and Failure Analysis

Our surrogate representation is metric-agnostic, as its construction does not depend on the target performance metric. As reported in Appendix Table 5, TimeDPE achieves consistent improvements over HMT on both range-aware metrics (VUS-PR: 40.2%; Range-F1: 26.4%) and conventional point-wise metrics (AUC-PR: 36.7%; AUC-ROC: 29.4%).

Additionally, we have updated Appendix Table 4 to include 95% confidence intervals for each data source. For a discussion of the failure cases, please refer to our response to Q3.

Thanks for the detailed response and the additional experiments!

Most of my concerns have been addressed. In particular, the TSB-AD-M results with the 22 score-based signals (23.0% Macro MAE reduction over HMT, better on 46/56 source–detector pairs) convincingly support applicability to multivariate settings. The adapted SATzilla/UReg comparison under the same DPE protocol also clarifies the relation to unsupervised model selection. The few-shot, stability, and latency analyses are helpful as well.

I still see the methodological novelty as moderate, and the remaining limits (only 22/29 signals for MTS, uneven cross-source gains, no reliable deployment-time fallback) should be stated more explicitly.

Overall, I will keep my original rating at 3.

Reviewer_M8wdOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses the problem of deployment-time performance estimation. The authors define it as the estimation of TSAD detectors performance on unlabeled unseen time series relying exclusively on previous performance on existing labelled dataset. Their method referred to as TimeDPE is to learn the relation between the two representation of the raw time series and its score on one hand and the detector performance on the other hand. Following this method, the authors claim that they decrease by 40.2% the macro MAE of instance-level VUS-PR estimation compared to existing strategies predicting the metric based on historical mean transfer (HMT) performances.

The experiments of the paper are lead on the sources of the TSB-AD-U dataset with more than 4 instances, it remains 860 time series from 17 different sources for analysis. The paper shows that the representations considered are central in the sample efficiency of TimeDPE as the alternatives perform worst in label scarcity settings. The complementarity of this representation is highlighted by the fact that different surrogate signals impact the results depending on the source-detector pair and is also emphasized by the Spearman correlation of 0.98 between the empirical surrogate sufficiency and the gain over HMT when the surrogate selected change.

Critical strengths
  1. To the best of my knowledge this work is novel and relevant for the TSAD community and this paper appears as a natural continuity of the works of Pouget et al, 2025 on perdicting deployment performance of image classification models.

  2. The method is based on shallow machine learning and surrogate signals are "simple" and allow for a deeper understanding of the detectors and sources, especially throught Figure 5)a).

  3. The experimentation is rigorous and the ablation study is informative on the gain of each component.

Critical weaknesses
  1. According to my understanding of the paper TimeDPE has not been estimated on multivariate times series. Univariate time series are already informative but more realistic scenarios would recquire the method to be applied on multivariate settings.

  2. Changing Figure 3 would be beneficial to the expressivity of the results. In low and high performing regions I aggree it forms clusters in the diagonal but it is not clear in the [0.2, 0.8] region. If ou keep a general sentence "Overall, TimeDPE predictions cluster more tightly around the diagonal representing perfect estimation than HMT predictions do." (lines 352-353) I believe you should better highlight it in the figure in the [0.2,0.8] region.

Questions for authors
  1. As a clarification, how do you adapt to the multivariate setting and what could be the limitations of your method ?

  2. Could you adjust the representation of the results in Figure 3 in an updated version ?

2 comments

We sincerely thank the reviewer for recognizing the novelty, relevance to the TSAD community, and experimental rigor of our work. We also appreciate the constructive suggestions regarding the extension to multivariate time series and the improvement of Figure 3. We have addressed both in our revised manuscript, as detailed below.

Response to W1 & Q1

We initially prioritized univariate time-series (UTS) because TSB-AD-U provides a broader range of instances per source (1–259 vs. 1–28 in TSB-AD-M), enabling a more comprehensive analysis of how historical sample size affects TimeDPE's performance. Nevertheless, we agree on the importance of multivariate time-series (MTS) validation and have conducted additional experiments accordingly.

MTS adaptation and limitations. Of TimeDPE's 29 surrogate signals, 22 rely solely on anomaly-score sequences and are directly applicable to MTS. The limitation is that the remaining seven raw–score relational signals were designed for UTS and cannot be applied directly to multichannel inputs. Extending them to MTS requires aggregating multiple channels while preserving cross-channel information, which is nontrivial.

Additional experiments and results. We evaluate TimeDPE with the 22 readily applicable surrogate signals and the original GP-RQ estimator on TSB-AD-M, using 7 detectors across 5 seeds under the same LOO protocol as used for TSB-AD-U. The main evaluation covers 8 sources (157 instances, ≥10 instances per source).

TimeDPE reduces Macro MAE from 0.1091 (HMT) to 0.0840 (TimeDPE), a 23.0% reduction, outperforming HMT in 46/56 source–detector pairs. All 8 sources and 7 detectors show positive macro-level gains, although improvements vary across sources. These results support TimeDPE's applicability to MTS, even without the 7 relational signals.

The full source–detector results of the MTS experiment are provided in Appendix Table 6 of the revised manuscript.

Response to W2 & Q2

We thank the reviewer for pointing out the limited clarity of Figure 3 in the intermediate detector-performance region. We have revised Figure 3 by adding a performance-stratified bar showing the relative reduction in Macro MAE (bin width = 0.1) and refined the accompanying statement accordingly. The revised Figure 3 is now included in the updated manuscript.

The revised figure supports our overall observation: TimeDPE outperforms HMT in 9 out of 10 performance intervals, which together contain 95.58% of all 8,600 evaluated detector–instance cases. The relative improvements reach 57.02% in [0, 0.1) and 52.88% in [0.9, 1.0], with these two extreme intervals jointly covering 4,610 cases (53.60% of all cases). Meanwhile, TimeDPE performs comparably to HMT in the intermediate region [0.4, 0.6), which contains only 808 cases (9.40%). This is expected: since HMT predicts a mean of historical performance, its errors are smallest within this intermediate region. This leaves little headroom for improvement in this region. Additionally, the relatively sparse historical observations may contribute to TimeDPE's weaker gains, consistent with our failure-case analysis highlighting the impact of limited historical supervision. Together, these factors may help explain the pattern observed in revised Figure 3.

I thank the authors for their detailed and constructive rebuttal. The additional experiments on TSB-AD-M (Table 6) address my main concern: the 23.0% Macro MAE reduction over HMT with only the 22 score-based signals supports the applicability of TimeDPE to multivariate settings. I understand that the limitation regarding the relational signals recquire additionnal work, this first experiment is an encouraging signal.

The revised Figure 3 with the stratified performance gain bar makes the results easier to interpret.

My concerns have been addressed, and I maintain my positive assessment of the paper.

Comments on the paper

1 comment

Summary of Rebuttal Discussions

We sincerely thank all reviewers for their constructive feedback and for recognizing the value of our DPE problem formulation and the rigor of our empirical evaluation. Their valuable suggestions have helped us further strengthen our work, with the following improvements incorporated into the revised manuscript:

  1. Reviewer M8wd and Reviewer uFcT suggested extending TimeDPE to multivariate time series, leading to additional experiments that further support its generalizability.

  2. Reviewer M8wd suggested improving Figure 3, helping us present the effectiveness of TimeDPE more clearly across different performance ranges.

  3. Reviewer uFcT suggested comparisons with unsupervised model-selection baselines, helping clarify the distinctions and potential complementarity between DPE and model selection.

  4. Reviewer fpx1 raised concerns about computational efficiency, motivating additional complexity and inference latency analyses that help clarify the practical deployment considerations of TimeDPE.

Finally, we would like to emphasize that even within the same data source, time-series instances can exhibit substantial variability, leading to large differences in detector performance, as illustrated in Figure 1. This variability already poses a significant challenge to reliable TSAD deployment and is precisely what motivates our DPE problem formulation.

Cross-source transfer goes beyond our primary within-source deployment setting, and our additional experiments were intended to explore the applicability boundaries of TimeDPE under substantial distribution shift.

Regarding generalizability across data sources, we have extensively evaluated TimeDPE on 17 diverse data sources from the widely used TSB-AD-U benchmark, covering at least six broad application domains, demonstrating its effectiveness across diverse within-source deployment settings.

We sincerely appreciate the reviewers' time and constructive suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.