acceptodds
Under review as a conference paper at ICLR 2027

Learning When to Adapt to Concept Drift from a Single Predictive Trajectory

12 views

Abstract

Concept drift detectors typically trigger adaptation when the detector statistic crosses a predefined threshold. While the threshold controls detector sensitivity, the ultimate goal of data stream learning is to maintain predictive performance under drift. Accordingly, a threshold adjustment method has been proposed to adjust the detector threshold according to the performance of the predictor. However, the existing method determines whether to adapt and adjusts the detector threshold by comparing the predictive performance of different thresholding schemes using future observations. This can be impractical in many real-world streaming settings, where threshold adjustment and adaptation decisions must be made online as data arrive. To address this limitation, we propose a single-trajectory framework that jointly adjusts the detector threshold and determines whether to adapt using information observed along the realized predictive trajectory. This allows the threshold to be adjusted and adaptation decisions to be made online without waiting for future observations. We provide an optimization interpretation of our threshold adjustment rule, analyze its stability, and derive a dynamic regret bound. Extensive experiments across multiple classifiers, drift detectors, datasets, and adaptation settings show that the proposed framework achieves predictive performance comparable to the state of the art methods while reducing computation overhead.

Primary area:
learning on time series and dynamical systems
Keywords:
Concept Drift, Data Stream Learning, Online Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3aNWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to determine when to adapt to concept drift without relying on future observations or multiple predictive trajectories. It proposes STAC, a single-trajectory framework that jointly adjusts drift detection thresholds and controls adaptation decisions using observed predictive performance. The paper also provides an online optimization interpretation, stability analysis, and a dynamic regret bound.

Critical strengths
  1. The paper addresses a well-motivated and practical problem: existing performance-guided threshold adjustment methods may require future observations and multiple predictive trajectories, limiting the applicability in online settings.
  2. The single-trajectory design is simple and efficient. Jointly adjusting detection thresholds and confirming adaptation based on predictive performance is a reasonable approach that can be integrated with existing drift detectors.
  3. The theoretical analysis is carefully developed, providing an optimization interpretation of the threshold update, stability analysis, and a dynamic regret guarantee.
  4. The experiments cover multiple classifiers, drift detectors, and datasets, including a more complex image-based setting. The results generally support the effectiveness and computational efficiency of the proposed framework.
Critical weaknesses
  1. The online setting relies on timely performance feedback.STAC requires observed predictive accuracy to update thresholds and confirm adaptation. However, labels may be delayed or unavailable in practical streaming applications, which could limit its applicability.
  2. The connection between the theoretical guarantees and actual adaptation performance is kind of indirect. The dynamic regret bound applies to a surrogate threshold objective rather than predictive performance. Moreover, the adaptation confirmation rules remain largely heuristic, particularly the use of post-adaptation performance changes to estimate the benefit of previous adaptations.
  3. The evaluation on complex data streams could be further clarified. On CIFAR-10-CD, the original STAC requires an additional persistence rule to outperform DTD. Besides, adaptation and ordinary updates use different training budgets, making it somehow difficult to isolate the contribution of adaptation timing.
Questions for authors
  1. How would STAC perform when labels are delayed or only partially available? Could the authors discuss whether the framework can accommodate delayed performance feedback?
  2. The accumulated performance improvement gtg_t is based on observed changes following previous adaptations. Since these changes may also reflect evolving data distributions, how reliably does gtg_t capture the actual benefit of adaptation? Could the authors provide some empirical evidence?
  3. Could the authors further clarify how the surrogate dynamic regret guarantee relates to predictive performance or adaptation quality? Even a brief discussion of this distinction would be helpful.
  4. On CIFAR-10-CD, why does an adaptation event use 5 training epochs while an ordinary update uses 10? Could the authors clarify whether the performance advantage remains under matched training budgets, and discuss the role of the additional persistence rule?
  5. Figure 1 currently focuses more on presenting the computational steps and mathematical notation than on explaining the intuition behind the framework. Could the authors improve Figure 1 to more clearly illustrate the roles of the different performance signals and how they contribute to threshold adjustment and adaptation decisions? A more intuitive visualization of the single-trajectory feedback mechanism would be helpful.
5 comments
AuthorsOpenReview ↗

Thank You for Your Thoughtful Review! We Will Provide a Detailed Response Soon.

Dear Reviewer 3aNW,

I would like to express my sincere gratitude for your insightful and constructive comments! We greatly appreciate the time and effort you devoted to reviewing our paper, as well as your positive assessment of our work.

We are currently preparing additional experiments and analyses to address your concerns. While we are still consolidating some of the results, we would like to share a brief outline of what we are preparing. We will provide a more detailed, point-by-point response once everything is ready:

  • In response to Weakness 1 and Question 1 (Delayed and partial labels). We will present additional experiments exploring how STAC can be integrated into an existing stream learning framework with delayed labels.

  • In response to Weakness 2 and Questions 2–3 (Adaptation feedback and theoretical guarantees). We will provide additional empirical analyses of gtg_t, clarify its role in adaptation decisions, and further explain the relationship between our surrogate dynamic regret guarantee and predictive performance.

  • In response to Weakness 3 and Question 4 (CIFAR-10-CD evaluation). We will further clarify the rationale behind our training configuration, supported by additional experimental evidence.

  • In response to Question 5 (Figure 1). We are working on revising Figure 1 to provide a more intuitive illustration of the STAC framework.

This outline may change slightly as we prepare our detailed responses.

Thank you again for your thoughtful review and encouragement. We would be grateful for your patience while we prepare our detailed response!

AuthorsOpenReview ↗

Response to Reviewer 3aNW (1/3)


Comment 1 (for Weakness 1 and Question 1): Additional Experiment on Delayed Labels

“How would STAC perform when labels are delayed or only partially available? Could the authors discuss whether the framework can accommodate delayed performance feedback?”

We thank the reviewer for highlighting the issue of delayed and partial labels. We believe STAC can be integrated into stream learning frameworks with delayed or partially available labels that incorporate concept drift detectors.

Here, we demonstrate one feasible way to apply STAC under delayed label feedback. Specifically, we integrate STAC into the TR-S framework proposed by Komorniczak et al. [1], while retaining its chunk-based processing (250 instances per chunk) and delayed-label protocol. When the labels of a previous chunk become available, we use the stored predictions to calculate performance feedback. STAC then adjusts the detection threshold and determines whether to adapt the predictor using the newly labeled chunk.

We evaluate STAC on three synthetic data streams (from the original study): synthetic_d5, synthetic_d10, and synthetic_d15, containing 5, 10, and 15 abrupt concept drifts, respectively. We also include four real-world data streams from our main experiments: Elec2, INSECTS-AB, INSECTS-GB, and INSECTS-IB. We consider four label delays (1, 10, 20, and 60 chunks), three classifiers, and eight drift detectors, with five random seeds per configuration.

The results show that:

  • STAC improves the average Balanced Accuracy over the original TR-S by 2.21, 5.27, and 2.68 percentage points for GNB, VFDT, and DNN, respectively.
  • Compared with Random, which matches the number of adaptations performed by STAC, the improvements are 0.92, 0.93, and 0.98 percentage points, respectively.

These results suggest that STAC can effectively adjust drift detector sensitivity and guide adaptation decisions to improve predictive performance under delayed label feedback.

Detailed results are provided in Section 6.7 and Appendix A.5 of the revised manuscript.


We apologize for being unable to address the next comment in this response due to the character limit. We will provide our response in a separate reply.


References

[1] Komorniczak, J., Ksieniewicz, P., & Zyblewski, P. (2026). Structuring the processing frameworks for data stream evaluation and application. Pattern Recognition, 172, 112516.

AuthorsOpenReview ↗

Response to Reviewer 3aNW (2/3)


Comment 2 (for Weakness 2 and Questions 2–3): Clarification and Additional Experiments

On surrogate regret and predictive performance.

We thank the reviewer for these important questions. We would like to clarify the connection between our surrogate objective and predictive performance by considering the following two scenarios:

  • When detector evidence qtq_t is strong but predictor evidence rtr_t is weak: The predictor has not experienced substantial performance degradation, but the detector is more likely to raise a drift alarm. This may lead to unnecessary adaptation, which can disturb the existing model and cause unstable predictive performance.

    • In this case, the detection threshold should be increased to reduce detector sensitivity.
  • When detector evidence qtq_t is weak but predictor evidence rtr_t is strong: The predictor has experienced substantial performance degradation, but the detector is less likely to raise a drift alarm. This may delay necessary adaptation and leave the predictor unable to respond promptly to concept drift.

    • In this case, the detection threshold should be decreased to increase detector sensitivity.

These two scenarios illustrate why aligning detector evidence with predictor evidence can help improve predictive performance. By adjusting the detection threshold to better reflect the predictor's adaptation needs, unnecessary adaptations could be reduced, and responses to performance degradation could become more timely.

To formalize this intuition, we define the surrogate-optimal threshold as θt∗=arg⁡min⁡θLt(θ)\theta_t^* = \arg\min_{\theta} L_t(\theta). Our surrogate objective can be equivalently expressed as a regularized KL divergence (please see the derivation below):

\begin{equation} \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad L_t(\theta) = \tau D_{\mathrm{KL}}(\mathrm{Bern}(r_t)|\mathrm{Bern}(q_t(\theta))) + \frac{\lambda}{2\eta}\theta^2 + C_t, \end{equation}

where CtC_t is independent of θ\theta. Therefore, θt∗\theta_t^* represents the threshold that aligns detector evidence qt(θ)q_t(\theta) with predictor evidence rtr_t, while regularization helps maintain stable threshold adjustments.

Our dynamic regret bound shows that STAC can track these time-varying surrogate-optimal thresholds in terms of cumulative surrogate loss. This provides a theoretical foundation for adjusting detector sensitivity according to the predictor's adaptation needs, which may help improve adaptation timing and predictive performance.


Derivation. Let qt(θ)=σ((S~t−θ)/τ)q_t(\theta)=\sigma((\widetilde S_t-\theta)/\tau) and z=(S~t−θ)/τz=(\widetilde S_t-\theta)/\tau. Treating rtr_t and S~t\widetilde S_t as fixed, the KL divergence between two Bernoulli distributions is:

\begin{equation} \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad D_{\mathrm{KL}}(\mathrm{Bern}(r_t)|\mathrm{Bern}(q_t(\theta)))=\log(1+e^z)-r_tz-H(r_t), \end{equation}

where H(r)=−rlog⁡r−(1−r)log⁡(1−r)H(r)=-r\log r-(1-r)\log(1-r) is the binary entropy, with 0log⁡0=00\log 0=0. Rearranging gives:

\begin{equation} \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad L_t(\theta)=\tau D_{\mathrm{KL}}(\mathrm{Bern}(r_t)|\mathrm{Bern}(q_t(\theta)))+\frac{\lambda}{2\eta}\theta^2+C_t, \end{equation}

where Ct=rtS~t+τH(rt)C_t=r_t\widetilde S_t+\tau H(r_t) is independent of θ\theta.

Differentiating yields ∇Lt(θ)=ληθ+rt−qt(θ)\nabla L_t(\theta)=\frac{\lambda}{\eta}\theta+r_t-q_t(\theta). Applying gradient descent with step size η\eta at θt\theta_t, where qt=qt(θt)q_t=q_t(\theta_t), gives θt+1=(1−λ)θt+η(qt−rt)\theta_{t+1}=(1-\lambda)\theta_t+\eta(q_t-r_t), which is exactly the threshold update of STAC.


On the reliability of gtg_t.

The purpose of gtg_t is to accumulate observed performance changes following previous adaptations, rather than estimate the exact benefit of an individual adaptation. We therefore conducted an additional experiment to examine whether the observed performance changes used to update gtg_t are associated with the benefits of adaptation relative to no adaptation.

For each adaptation event, we compared two versions of the same model: one with adaptation and one without adaptation. We evaluated both on the same next data chunk and compared their accuracy difference with the observed performance change used to update gtg_t.

We conducted 720 experiments across eight datasets, six drift detectors, three classifiers, and five random seeds, obtaining 10,755 valid comparisons. Across the pooled adaptation events, the Pearson correlation coefficients were 0.969, 0.923, and 0.386 for GNB, VFDT, and DNN, respectively. These results show a strong overall correlation between the observed performance changes used to update gtg_t and the benefits of adaptation relative to no adaptation for GNB and VFDT, while the correlation is weaker for DNN.

Together, the theoretical analysis explains how STAC adjusts detector sensitivity according to predictor evidence, while the additional experiment provides empirical evidence for the performance feedback underlying gtg_t.

Reviewer_qTZYOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In this work, the authors introduce STAC which is a meta-adaptation approach which uses drift detection signals, in combination with a deployed stream learner's historical performance to learn "when to adapt" instead of merely adapting when a drift detection event is signalled as is typically done in stream learning literature.

The authors validate STAC by comparing it with fixed detector adaptation policies, and DTD another related method. The authors also demonstrate that STAC is detector and model agnostic.

NOTE: The main body of the review is unchanged, but the score has been updated to reflect the discussion with the authors during the rebuttal.

Critical strengths
  1. The paper addresses a relevant problem in concept drift adaptation research. In general, I like the idea that model adaptation is something that should be learned, not fixed like is commonly done in related work.
Critical weaknesses

For the most part, I have serious concerns about the evaluation process of this work. Namely:

  1. Potential Data Leakage:

In Section A.4.5, the authors show that their default configuration seems to be the set of parameters that perform reasonably across all predictive models investigated. The authors do not state exactly how the default configuration was determined, but I find it highly unconvincing that the default configuration was determined without tuning STAC on the data streams evaluated in this work. This is effectively data leakage (the other approaches used default values), and biases the results to the point where I don't find them reliable. This is a fundamental evaluation error, and is the primary reason for the clear reject.

  1. Missing baselines and Outdated Adaptation Policies:

While this work covers several model architectures, data streams, and drift detectors, they are quite outdated or missing certain baselines. In truth, the evaluation setup in this work mimics PUDD, and DTD a little too closely in my opinion. For missing architectures, the paper should evaluate an ensembling strategy (such as Streaming Random Patches) which are state-of-the-art over simple Naive Bayes and Hoeffding Trees.

The paper should also consider additional, modern, data streams such as the Insects, Asfault, Rialto, etc. (which you can get from the USP DS repository) and rely less on synthetic data for the type of evaluation they conducted in this work.

For the drift detectors, it is odd that three versions of PUDD are included yet the authors neglect to include ADWIN (arguably the most commonly used drift detector) and methods that have proven to be generally effective such as SEED and STEPD (see [1]).

Lastly, the simple reset strategy the authors use for the adaptation policies for the GNB and VFDT do not reflect how modern stream learning approaches use drift detectors as signals for adaptation policies. Techniques such as the Hoeffding Adaptive Tree and most stream ensembles will either use two detectors (or a detector with two thresholds such as DDM) to trigger warning and reset events. When a warning event is triggered a background model is trained, and the when a reset event is triggered, the background model replaces the active model. This seems like a much more appropriate baseline comparison to the authors method.

In general, this work does not engage sufficiently with the state-of-the-art. I believe a significant overhaul in the evaluation is needed to make this work worthy of acceptance.

[1] Cerqueira, V., Gomes, H. M., Heyden, M., Pfahringer, B., & Bifet, A. (2026, August). A Framework for Evaluating and Benchmarking Concept Drift Detection Methods. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2 (pp. 8671-8682).

Questions for authors

Questions:

  1. Potential Data Leakage: Could you confirm if my suspicions of data leakage are true? In particular, I would need to know more about how the default configuration for STAC was chosen.

  2. Erroneous Runtimes: The authors report in Tables 9-11 that STAC can sometimes outperform the runtime of the Fixed detectors. Is this a reporting error? The detectors themselves are required for STAC so it would make sense to me that the runtime of STAC is lower-bounded by the runtime of the detector. Could you elaborate on these results?

Comments:

  1. I would ask the authors to consider if their evaluation setup is appropriate for STAC. I know the evaluation mimics DTD's, but what are you actually trying to show? If, as I understood it, STAC is meant enable adaptations that are actually useful, it would make sense to evaluate that explicitly (instead of implying that simply showing performance improvement means that STAC is working as intended). Table 2, with its reporting of the number of adaptations is promising for example. Perhaps future work could explicitly consider (synthetic) settings where the timing of adaptations could be explicitly measured.

  2. I would also ask the authors to consider a broader range of related work. Other than the missing baselines which I pointed out, there are other aspects of this paper that don't necessarily reflect current trends in Stream Learning. For example, Cohen's Kappa statistic is considered a more robust evaluation metric than Accuracy. Missing / Delayed labels are evaluation settings which are receiving renewed focus (and are of relevance to this paper). The mini-batch test-then-train approach to evaluation is also abnormal when evaluating the incremental learners (GNB, VFDT), and should be motivated a bit better. Regression is also missing completely from this work, and there have been renewed efforts to focus more on regression tasks in stream settings (see SOKNL in capymoa for example).

7 comments
AuthorsOpenReview ↗

Thank You for Your Constructive Feedback, We Are Working Hard at Preparing a Detailed Response!

Dear Reviewer qTZY,

I would like to express my gratitude for your insightful comments. We did learn a great deal from them!

We are currently working on several additional experiments to address your concerns. While some experiments are still running, we would like to share a brief outline of what we are preparing. We will provide a more detailed, point-by-point response once the results are ready:

  • In response to Weakness 1 and Question 1 (Hyperparameter selection). We will clarify how we originally selected our hyperparameters and report our main experiments rerun using a more rigorous leave-one-dataset-out (LODO) evaluation protocol.

  • In response to Weakness 2 (Additional experiments). We will provide additional results on modern datasets and drift detectors.

  • In response to Question 2 (Runtime differences). We will explain why STAC can sometimes be faster than Fixed, despite the additional computation involved.

  • In response to Comment 1 (Motivation and evaluation). We will further explain what STAC aims to achieve and include additional evaluation metrics, such as Macro-F1, Cohen's Kappa, and the number of adaptations.

  • In response to Comment 2 and RC1 (Background and Foreground Training). We will explain our training protocol and provide additional experiments with background and foreground training.

  • In response to Comment 2 and RC2 (Instance-wise Stream Learning). We provide additional experiments under an instance-wise stream learning protocol.

  • In response to RC3 and RC4 (Data Leakage and Dataset Characteristics). We appreciate your further clarifications regarding the potential data leakage in synthetic streams and the characteristics of the INSECTS datasets. We will revise the relevant manuscript descriptions accordingly.

  • In response to Weakness 2 (Additional Experiments in Extended Settings). We further explore the applicability of STAC to ensemble learning and delayed label feedback.

This outline may change a little as we prepare our detailed responses.

Thank you again for your review. We would be grateful for your patience while we prepare our detailed response!

AuthorsOpenReview ↗

Response to Reviewer qTZY (1/6)


Comment 1 (for Weakness 1 & Question 1): Clarification and Additional Experiments.

"The authors do not state exactly how the default configuration was determined."

"Could you confirm if my suspicions of data leakage are true? In particular, I would need to know more about how the default configuration for STAC was chosen."

Thank you very much for pointing out this issue. We agree that our original paper did not clearly explain how the default hyperparameters were selected, and we understand your concerns about potential data leakage.

We evaluated different hyperparameter settings across different datasets, drift detectors, and predictive models, using a different set of random seeds from those used in our main experiments. We examined each hyperparameter separately and combined the values that showed relatively stable performance into a single default configuration, rather than performing an exhaustive grid search.

We agree that our original hyperparameter selection procedure could have limitations:

  • For synthetic datasets, this seemed reasonable because different random seeds generated different data streams.
  • For real-world datasets, however, changing the random seeds did not change the underlying data streams. Models initialized with different random seeds were still trained and evaluated on the same data streams, which could introduce selection bias.

To address this concern, we conducted additional experiments using a leave-one-dataset-out (LODO) protocol, following the recommendation of Cerqueira et al. [1].

For the LODO evaluation, we used the original eight datasets. Each time, we held out one dataset and selected the hyperparameters based on the average accuracy of the remaining seven datasets. We selected a single configuration for each held-out dataset, shared across all predictive models, drift detectors, and both training modes. We then evaluated the selected configurations using 20 random seeds.

The results are summarized below:

Table 1: Accuracy (%) on the eight datasets with LODO evaluation (20 random seeds).

DatasetFixedDTDSTAC (LODO)STAC − DTD
Airline57.8959.3759.74+0.38
Elec273.0872.5470.95-1.59
PowerSupply71.2071.3671.42+0.05
SEA095.2095.2995.92+0.63
SEA1086.6887.0687.44+0.38
SEA2076.9677.2977.57+0.28
SINE87.1887.0888.15+1.06
MIXED85.4584.7485.80+1.06
Average79.2079.3479.62+0.28

The results show that:

  • Predictive performance. With LODO, STAC achieved an average accuracy of 79.62%, compared with 80.08% in our original results. However, STAC still remains competitive with DTD (79.34%) and Fixed (79.20%). STAC also outperforms DTD on 7 out of 8 datasets.
  • Computational efficiency. On average, STAC requires only 1.067× the runtime of Fixed, while DTD requires 2.013×. This means DTD takes about 1.89× as long as STAC.

All the results and newly added citations have been updated in the revised manuscript.


References

[1] Vitor Cerqueira, Heitor Murilo Gomes, Marco Heyden, Bernhard Pfahringer, and Albert Bifet. A Framework for Evaluating and Benchmarking Concept Drift Detection Methods. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 8671–8682, 2026.

AuthorsOpenReview ↗

Response to Reviewer qTZY (2/6)


Comment 2 (for Weakness 2): Additional Datasets and Drift Detectors.

"The paper should also consider additional, modern, data streams such as the Insects, Asfault, Rialto, etc."

"For the drift detectors, it is odd that three versions of PUDD are included yet the authors neglect to include ADWIN [...] and methods [...] such as SEED and STEPD."

Thank you for these suggestions! We have extended our evaluation with three additional real-world data streams from INSECTS [1]: Abrupt Balanced (AB), Gradual Balanced (GB), and Incremental Balanced (IB), to better examine STAC's performance under different real-world drift patterns.

Meanwhile, following your suggestion, we replaced PUDD-1 and PUDD-5 (which were already evaluated in DTD) with two additional drift detectors: ADWIN [2] and STEPD [3], while retaining the more balanced PUDD-3.

We evaluated all three new datasets and the additional drift detectors under the same evaluation protocol, using 20 random seeds. For the INSECTS datasets, we followed the same procedure, treating the INSECTS streams as additional held-out datasets and selecting hyperparameters from the original eight datasets (Airline, Elec2, PowerSupply, SEA0, SEA10, SEA20, SINE, and MIXED).

The results are summarized below:

Table 1: Accuracy (%) on additional datasets and drift detectors (20 random seeds).

(a) Additional real-world datasets (8 drift detectors)

DatasetFixedDTDSTACSTAC − DTD
INSECTS-AB63.3964.4864.65+0.17
INSECTS-GB62.7366.6366.28-0.35
INSECTS-IB55.0655.7157.18+1.47
Average60.3962.2862.71+0.43

(b) Additional drift detectors (11 datasets)

DetectorFixedDTDSTACSTAC − DTD
ADWIN74.0674.9075.15+0.25
STEPD74.2774.9574.99+0.04

The results show that:

  • Additional datasets. STAC achieves higher average accuracy than Fixed and DTD across the three INSECTS streams, although it does not outperform DTD on every individual stream.
  • Additional drift detectors. STAC also remains competitive with DTD when using ADWIN and STEPD, achieving higher average accuracy with both detectors.

Comment 3 (for Question 2): Clarification of Runtime Differences.

"The authors report in Tables 9-11 that STAC can sometimes outperform the runtime of the Fixed detectors. Is this a reporting error? The detectors themselves are required for STAC so it would make sense to me that the runtime of STAC is lower-bounded by the runtime of the detector. Could you elaborate on these results?"

Thank you so much for carefully reading our paper and raising this question! We report the end-to-end runtime of each method under the same experimental settings, including prediction, drift detection, and model updates. This allows us to compare their overall computational costs, while also accounting for differences in their adaptation decisions.

Theoretically, STAC only adds a few computations per chunk for threshold adjustment and adaptation decisions, with an additional time complexity of O(T), where T is the number of chunks.

In practice, adaptation decisions can also affect the overall runtime in two ways:

  • Model training: Different adaptation decisions lead to different training trajectories. For example, adapting VFDT resets the decision tree, which may change its size and the cost of subsequent predictions and updates. For DNN, adaptation reinitializes the output layer while retaining the hidden layers, whereas regular updates continue training the existing model.

  • Drift detection: Adaptation can also affect the detector's computational cost. For example, in PUDD, adaptation clears the accumulated history, reducing the amount of data involved in subsequent statistical comparisons. In such cases, more frequent adaptations may also reduce the overall runtime.

Overall, STAC takes 1.067× the runtime of Fixed, compared with 2.013× for DTD. Since adaptation decisions affect both model training and drift detection, STAC can sometimes run faster than Fixed, even with its additional computations.

We have also added the adaptation counts to Appendix A.4.4 of the revised manuscript to provide more context for the runtime comparisons.


References

[1] Vinicius M. A. Souza, Denis M. dos Reis, Andre G. Maletzke, and Gustavo E. A. P. A. Batista. Challenges in Benchmarking Stream Learning Algorithms with Real-world Data. Data Mining and Knowledge Discovery, 34(6):1805–1858, 2020.

[2] Albert Bifet and Ricard Gavaldà. Learning from Time-Changing Data with Adaptive Windowing. In Proceedings of the 2007 SIAM International Conference on Data Mining, pp. 443–448, 2007.

[3] Kyosuke Nishida and Koichiro Yamauchi. Detecting Concept Drift Using Statistical Testing. In Proceedings of the 10th International Conference on Discovery Science, pp. 264–269, 2007.

Reviewer_UUdTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Single-Trajectory Threshold Learning and Adaptation Control (STAC), an online framework for concept drift adaptation that jointly adjusts drift detection thresholds and determines whether model adaptation is necessary using information from a single realized predictive trajectory. STAC updates detection thresholds based on the discrepancy between drift evidence and predictive performance degradation, while its adaptation policy incorporates recent performance deterioration and historical improvements following previous adaptations. The authors provide an online optimization interpretation of the threshold update, analyze its stability, and derive a dynamic regret bound. Experiments across multiple classifiers, drift detectors, and datasets demonstrate that STAC achieves competitive predictive performance while reducing computational overhead compared with existing threshold adaptation methods, particularly DTD.

Critical strengths

S1. The paper proposes a single-trajectory framework that jointly adjusts detection thresholds and controls model adaptation without requiring future observations or multiple predictive trajectories. This design improves the practicality and computational efficiency of online adaptation.

S2. Extensive experiments across diverse classifiers, drift detectors, and datasets demonstrate competitive predictive performance with reduced computational overhead compared with DTD.

Critical weaknesses

W1. The threshold adjustment and adaptation rules rely on heuristic combinations of performance indicators, with limited justification for their general effectiveness across different drift patterns. Moreover, the theoretical guarantees concern a surrogate objective rather than actual predictive performance.

W2. The CIFAR-10-CD evaluation uses different per-chunk training budgets depending on adaptation decisions, confounding the attribution of performance gains to the proposed method. This weakens the reliability of the experimental conclusions regarding adaptation effectiveness.

W3. STAC relies on observed predictive performance to update its threshold and determine whether adaptation is warranted, implicitly requiring timely access to ground-truth labels. This assumption limits its applicability to streaming environments with delayed, sparse, or unavailable labels, which are common challenges in practical online learning settings.

Questions for authors

None.

5 comments
AuthorsOpenReview ↗

Preliminary Response to Reviewer UUdT

Dear Reviewer UUdT,

We sincerely thank you for your thoughtful and constructive review. We greatly appreciate the time and effort you devoted to evaluating our work. Your comments have highlighted several important aspects that warrant further clarification and additional empirical evidence.

We are currently preparing a detailed point-by-point response and conducting additional analyses to address your concerns more thoroughly. While this work is still ongoing, we would like to first provide a brief outline of how we plan to address your comments:

  • In response to Weakness 1 (part 1). We have conducted additional experiments to evaluate the effectiveness of STAC across different drift patterns, including abrupt, gradual, and incremental drift.

  • In response to Weakness 1 (part 2). We will further explain the motivation behind our theoretical analysis.

  • In response to Weakness 2. We will further clarify the rationale behind our training configuration, supported by additional experimental evidence.

  • In response to Weakness 3. We will clarify the scope of our work and provide additional expirements.

We sincerely thank you again for your time and constructive feedback! We will continue to update our responses and revise the manuscript accordingly.

AuthorsOpenReview ↗

Response to Reviewer UUdT (1/3)


Comment 1 (for Weakness 1): Additional Experiments on Different Drift Patterns.

"The threshold adjustment and adaptation rules rely on heuristic combinations of performance indicators, with limited justification for their general effectiveness across different drift patterns."

We have conducted additional experiments using the Rotating Hyperplane generator [1] to simulate abrupt, gradual, and incremental drift [2]. These experiments allow us to examine how STAC responds to different drift patterns.

The results show that:

  • STAC responds more actively around abrupt drift points and during gradual drift transitions.
  • STAC achieves higher average accuracy than both the fixed-threshold baseline (Fixed) and the existing dynamic threshold adjustment method (DTD) across all three drift patterns.
  • STAC also outperforms Random, which performs the same number of adaptations as STAC but at randomly selected times.

The new results are highlighted in blue in Section 6.3 and Appendix A.4.2 of the revised manuscript.


References

[1] Geoff Hulten, Laurie Spencer, and Pedro Domingos. Mining Time-Changing Data Streams. In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 97–106. ACM, 2001.

[2] Heitor M. Gomes, Albert Bifet, Jesse Read, Jean Paul Barddal, Fabrício Enembreck, Bernhard Pfahringer, Geoff Holmes, and Talel Abdessalem. Adaptive Random Forests for Evolving Data Stream Classification. Machine Learning, 106(9):1469–1495, 2017.

AuthorsOpenReview ↗

Response to Reviewer UUdT (2/3)


Comment 2 (for Weakness 1): Additional Mathematical Justification and Evidence from Experiments.

"Moreover, the theoretical guarantees concern a surrogate objective rather than actual predictive performance."

In online stream settings, future predictive performance is unavailable when adaptation decisions are made. Even if we had access to the entire data stream, there would be 2T2^T possible adaptation decision sequences for TT chunks. Each decision affects subsequent model states and predictions, making exhaustive search among the 2T2^T sequences impractical. This motivates us to use a surrogate objective based only on observed information.

In our method, we use the discrepancy between detector evidence qtq_t and predictor evidence rtr_t to adjust the detection threshold. The connection between these two sources of evidence and predictive performance can be illustrated through two scenarios:

  • When qtq_t is strong but rtr_t is weak: The detector indicates potential drift, but the predictor has not experienced substantial performance degradation. This may lead to unnecessary adaptation. Increasing the detection threshold can help reduce such adaptations.
  • When qtq_t is weak but rtr_t is strong: The predictor has experienced substantial performance degradation, but the detector provides insufficient evidence of drift. This may delay necessary adaptation. Decreasing the detection threshold can help improve responsiveness.

These scenarios illustrate why aligning detector evidence with predictor evidence can help improve adaptation decisions and predictive performance.

To formalize this intuition, we define the surrogate-optimal threshold as θt∗=arg⁡min⁡θLt(θ)\theta_t^*=\arg\min_{\theta}L_t(\theta). Our surrogate objective can be equivalently expressed as a regularized KL divergence between two Bernoulli distributions corresponding to qtq_t and rtr_t, up to a constant independent of the threshold:

\begin{equation} \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad L_t(\theta)=\tau D_{\mathrm{KL}}(\mathrm{Bern}(r_t)|\mathrm{Bern}(q_t(\theta)))+\frac{\lambda}{2\eta}\theta^2+C_t, \end{equation}

where CtC_t is independent of θ\theta. Minimizing this objective encourages alignment between detector evidence and predictor evidence while regularizing the threshold. Applying gradient descent to this objective yields exactly the threshold update rule of STAC.

Our dynamic regret bound characterizes STAC's ability to track these time-varying surrogate-optimal thresholds in terms of cumulative surrogate loss. Although this does not directly guarantee predictive accuracy, it provides a theoretical foundation for adjusting detector sensitivity according to the predictor's adaptation needs.

We also evaluate the actual predictive performance of STAC through experiments. The results show that STAC can outperform Fixed while achieving comparable predictive performance to DTD, which relies on future performance comparisons. In contrast, STAC only uses information observed so far.


Derivation. Let qt(θ)=σ((S~t−θ)/τ)q_t(\theta)=\sigma((\widetilde S_t-\theta)/\tau) and z=(S~t−θ)/τz=(\widetilde S_t-\theta)/\tau. Treating rtr_t and S~t\widetilde S_t as fixed, the KL divergence between two Bernoulli distributions is:

\begin{equation} \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad D_{\mathrm{KL}}(\mathrm{Bern}(r_t)|\mathrm{Bern}(q_t(\theta)))=\log(1+e^z)-r_tz-H(r_t), \end{equation}

where H(r)=−rlog⁡r−(1−r)log⁡(1−r)H(r)=-r\log r-(1-r)\log(1-r) is the binary entropy, with 0log⁡0=00\log 0=0. Rearranging gives:

\begin{equation} \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad L_t(\theta)=\tau D_{\mathrm{KL}}(\mathrm{Bern}(r_t)|\mathrm{Bern}(q_t(\theta)))+\frac{\lambda}{2\eta}\theta^2+C_t, \end{equation}

where Ct=rtS~t+τH(rt)C_t=r_t\widetilde S_t+\tau H(r_t) is independent of θ\theta.

Differentiating yields ∇Lt(θ)=ληθ+rt−qt(θ)\nabla L_t(\theta)=\frac{\lambda}{\eta}\theta+r_t-q_t(\theta). Applying gradient descent with step size η\eta at θt\theta_t, where qt=qt(θt)q_t=q_t(\theta_t), gives θt+1=(1−λ)θt+η(qt−rt)\theta_{t+1}=(1-\lambda)\theta_t+\eta(q_t-r_t), which is exactly the threshold update of STAC.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Review & Discussion Summary (2/2)

Reviewer qTZY (continued)

We then addressed the reviewer's follow-up concerns:

  • in response to concern 5: We added experiments under the background and foreground training paradigm. We also clarified that this paradigm was not included in our original evaluation to ensure a fair comparison with DTD (additional results are provided in Section 6.7 and Appendix A.5 of the revised manuscript; see comment 5).
  • in response to concern 6: We added experiments under an instance-wise stream learning protocol. We also clarified that this setting was not included in our original evaluation to ensure a fair comparison with DTD (additional results are provided in Section 6.7 and Appendix A.5 of the revised manuscript; see comment 6).
  • in response to additional comments: We acknowledged the issues raised by the reviewer and corrected the relevant descriptions in the revised manuscript (see comment 7).
  • additional experiments: We further explored the applicability of STAC to broader stream-learning settings, including ensemble learning and delayed-label feedback (additional results are provided in Section 6.7 and Appendix A.5 of the revised manuscript; see comment 8).

Reviewer 3aNW

We sincerely appreciate Reviewer 3aNW's constructive feedback and positive assessment of our work. We summarize the main concerns as follows:

  • concern 1: The reliance on timely performance feedback may limit the applicability of STAC in settings with delayed or partially available labels (Weakness 1 and Question 1).
  • concern 2: The reliability of the historical performance signal gtg_t in estimating the actual benefit of adaptation requires further empirical evidence (Weakness 2 and Question 2).
  • concern 3: A brief discussion clarifying the relationship between the surrogate dynamic regret guarantee and actual predictive performance or adaptation quality would be helpful (Weakness 2 and Question 3).
  • concern 4: The CIFAR-10-CD evaluation requires further justification of the different training budgets and the additional persistence rule (Weakness 3 and Question 4).
  • concern 5: Figure 1 should provide a more intuitive illustration of the STAC framework and its single-trajectory feedback mechanism (Question 5).

We provided detailed responses to the reviewer's concerns as follows:

  • in response to concern 1: We conducted additional experiments by integrating STAC into an existing stream learning framework to demonstrate its potential applicability under delayed label feedback (additional results are provided in Section 6.7 and Appendix A.5 of the revised manuscript; see comment 1).
  • in response to concern 2: We clarified the role of gtg_t and conducted additional experiments to evaluate the benefit of adaptation compared with no adaptation at each adaptation event. The results provide empirical evidence that the performance feedback underlying gtg_t is positively associated with adaptation benefits (see comment 2).
  • in response to concern 3: We clarified the distinction between surrogate regret and predictive performance, and provided additional mathematical justification to explain how our surrogate objective supports performance-guided threshold adjustment (see comment 2).
  • in response to concern 4: We clarified the rationale behind the CIFAR-10-CD training configuration, conducted additional budget-matched experiments to further support the reliability of our experimental conclusions, and explained the role of the additional persistence rule (additional results are provided in Section 6.2 and Appendix A.4.3 of the revised manuscript; see comment 3).
  • in response to concern 5: Following the reviewer's suggestion, we revised Figure 1 to provide a clearer and more intuitive illustration of the STAC framework (see the new Figure 1 of the revised manuscript; see comment 4).

We sincerely thank all reviewers and ACs again for their time and constructive feedback. We believe that the extensive additional experiments, strengthened mathematical justifications, and substantial manuscript revisions have directly addressed the reviewers' concerns and significantly strengthened the empirical and theoretical foundations of our work.

We hope these efforts and the additional evidence will be fully considered in the final decision.

AuthorsOpenReview ↗

Review & Discussion Summary (1/2)

We sincerely thank all reviewers and Area Chairs (ACs) for their invaluable service and constructive feedback. Below, we summarize the main concerns raised by each reviewer and how we addressed them during the rebuttal and discussion.

To sum up, we have provided point-by-point responses to all comments raised by the three reviewers and conducted additional experiments to support our responses to nearly all of their concerns.


Reviewer UUdT

We understand Reviewer UUdT's main concerns to be as follows:

  • concern 1: The effectiveness of STAC across different drift patterns has not been sufficiently demonstrated (Weakness 1).
  • concern 2: The connection between the theoretical guarantees on the surrogate objective and actual predictive performance requires further justification (Weakness 1).
  • concern 3: The different training budgets in the CIFAR-10-CD evaluation may affect the reliability of the experimental conclusions (Weakness 2).
  • concern 4: The applicability of STAC in settings with delayed, sparse, or unavailable labels has not been demonstrated (Weakness 3).

To address these concerns, we provided point-by-point responses:

  • in response to concern 1: We conducted additional experiments to demonstrate the effectiveness of STAC across different drift patterns (additional results are provided in Section 6.3 and Appendix A.4.2 of the revised manuscript; see comment 1).
  • in response to concern 2: We clarified why actual future predictive performance cannot be directly optimized in the online setting, and further provided mathematical justification to explain how our surrogate objective is related to predictive performance (see comment 2).
  • in response to concern 3: We clarified the rationale behind the CIFAR-10-CD training configuration and conducted additional budget-matched experiments to further support the reliability of our experimental conclusions (additional results are provided in Section 6.2 and Appendix A.4.3 of the revised manuscript; see comment 3).
  • in response to concern 4: We conducted additional experiments by integrating STAC into an existing framework to explore its potential applicability in these additional settings (additional results are provided in Section 6.7 and Appendix A.5 of the revised manuscript; see comment 4).

Reviewer qTZY

We sincerely appreciate Reviewer qTZY's valuable and constructive suggestions, which helped us identify and address important limitations in our original evaluation. We summarize the main concerns as follows:

  • concern 1: Potential data leakage arising from hyperparameter selection (Weakness 1 and Question 1).
  • concern 2: The evaluation should include more recent data streams and widely used drift detectors (Weakness 2).
  • concern 3: STAC's occasional runtime advantage over Fixed requires further explanation (Question 2).
  • concern 4: The motivation for STAC should be better explained, and additional performance metrics should be included (Comments 1 and 2).

Following our initial responses, Reviewer qTZY provided further feedback and highlighted several remaining concerns:

  • concern 5: The evaluation lacks modern adaptation mechanisms: the background and foreground training paradigm with warning and reset detectors (Weakness 2 and RC1).
  • concern 6: Instance-wise learning experiments should be conducted, as chunk-based learning does not reflect how the compared drift detectors were originally designed to be used (Comment 2 and RC2).
  • additional comments: The reviewer also raised additional comments regarding potential data leakage arising from synthetic drift settings and the characteristics of the INSECTS datasets (RC3 and RC4).

Subsequently, we carefully addressed the reviewer's concerns point by point:

  • in response to concern 1: We acknowledged the limitations of our original hyperparameter selection and adopted a more rigorous leave-one-dataset-out (LODO) protocol. We updated our main experimental results accordingly and explicitly discussed the differences from the original evaluation (see comment 1).
  • in response to concern 2: We substantially extended our evaluation by including additional modern data streams and widely used drift detectors (additional results are provided in Section 6 and Appendices A.3–A.5 of the revised manuscript; see comment 2).
  • in response to concern 3: We explained why STAC can sometimes run faster than Fixed, clarifying that this is a reasonable consequence of different adaptation decisions rather than a reporting error (see comment 3).
  • in response to concern 4: Following the reviewer's suggestions, we further clarified the motivation of STAC and included additional performance metrics for a more comprehensive evaluation (additional results are provided in Section 6 and Appendices A.3–A.5 of the revised manuscript; see comment 4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.