- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to determine when to adapt to concept drift without relying on future observations or multiple predictive trajectories. It proposes STAC, a single-trajectory framework that jointly adjusts drift detection thresholds and controls adaptation decisions using observed predictive performance. The paper also provides an online optimization interpretation, stability analysis, and a dynamic regret bound.
Critical strengths
- The paper addresses a well-motivated and practical problem: existing performance-guided threshold adjustment methods may require future observations and multiple predictive trajectories, limiting the applicability in online settings.
- The single-trajectory design is simple and efficient. Jointly adjusting detection thresholds and confirming adaptation based on predictive performance is a reasonable approach that can be integrated with existing drift detectors.
- The theoretical analysis is carefully developed, providing an optimization interpretation of the threshold update, stability analysis, and a dynamic regret guarantee.
- The experiments cover multiple classifiers, drift detectors, and datasets, including a more complex image-based setting. The results generally support the effectiveness and computational efficiency of the proposed framework.
Critical weaknesses
- The online setting relies on timely performance feedback.STAC requires observed predictive accuracy to update thresholds and confirm adaptation. However, labels may be delayed or unavailable in practical streaming applications, which could limit its applicability.
- The connection between the theoretical guarantees and actual adaptation performance is kind of indirect. The dynamic regret bound applies to a surrogate threshold objective rather than predictive performance. Moreover, the adaptation confirmation rules remain largely heuristic, particularly the use of post-adaptation performance changes to estimate the benefit of previous adaptations.
- The evaluation on complex data streams could be further clarified. On CIFAR-10-CD, the original STAC requires an additional persistence rule to outperform DTD. Besides, adaptation and ordinary updates use different training budgets, making it somehow difficult to isolate the contribution of adaptation timing.
Questions for authors
- How would STAC perform when labels are delayed or only partially available? Could the authors discuss whether the framework can accommodate delayed performance feedback?
- The accumulated performance improvement is based on observed changes following previous adaptations. Since these changes may also reflect evolving data distributions, how reliably does capture the actual benefit of adaptation? Could the authors provide some empirical evidence?
- Could the authors further clarify how the surrogate dynamic regret guarantee relates to predictive performance or adaptation quality? Even a brief discussion of this distinction would be helpful.
- On CIFAR-10-CD, why does an adaptation event use 5 training epochs while an ordinary update uses 10? Could the authors clarify whether the performance advantage remains under matched training budgets, and discuss the role of the additional persistence rule?
- Figure 1 currently focuses more on presenting the computational steps and mathematical notation than on explaining the intuition behind the framework. Could the authors improve Figure 1 to more clearly illustrate the roles of the different performance signals and how they contribute to threshold adjustment and adaptation decisions? A more intuitive visualization of the single-trajectory feedback mechanism would be helpful.
5 comments
Response to Reviewer 3aNW (1/3)
Comment 1 (for Weakness 1 and Question 1): Additional Experiment on Delayed Labels
“How would STAC perform when labels are delayed or only partially available? Could the authors discuss whether the framework can accommodate delayed performance feedback?”
We thank the reviewer for highlighting the issue of delayed and partial labels. We believe STAC can be integrated into stream learning frameworks with delayed or partially available labels that incorporate concept drift detectors.
Here, we demonstrate one feasible way to apply STAC under delayed label feedback. Specifically, we integrate STAC into the TR-S framework proposed by Komorniczak et al. [1], while retaining its chunk-based processing (250 instances per chunk) and delayed-label protocol. When the labels of a previous chunk become available, we use the stored predictions to calculate performance feedback. STAC then adjusts the detection threshold and determines whether to adapt the predictor using the newly labeled chunk.
We evaluate STAC on three synthetic data streams (from the original study): synthetic_d5, synthetic_d10, and synthetic_d15, containing 5, 10, and 15 abrupt concept drifts, respectively. We also include four real-world data streams from our main experiments: Elec2, INSECTS-AB, INSECTS-GB, and INSECTS-IB. We consider four label delays (1, 10, 20, and 60 chunks), three classifiers, and eight drift detectors, with five random seeds per configuration.
The results show that:
- STAC improves the average Balanced Accuracy over the original TR-S by 2.21, 5.27, and 2.68 percentage points for GNB, VFDT, and DNN, respectively.
- Compared with Random, which matches the number of adaptations performed by STAC, the improvements are 0.92, 0.93, and 0.98 percentage points, respectively.
These results suggest that STAC can effectively adjust drift detector sensitivity and guide adaptation decisions to improve predictive performance under delayed label feedback.
Detailed results are provided in Section 6.7 and Appendix A.5 of the revised manuscript.
We apologize for being unable to address the next comment in this response due to the character limit. We will provide our response in a separate reply.
References
[1] Komorniczak, J., Ksieniewicz, P., & Zyblewski, P. (2026). Structuring the processing frameworks for data stream evaluation and application. Pattern Recognition, 172, 112516.
Response to Reviewer 3aNW (2/3)
Comment 2 (for Weakness 2 and Questions 2–3): Clarification and Additional Experiments
On surrogate regret and predictive performance.
We thank the reviewer for these important questions. We would like to clarify the connection between our surrogate objective and predictive performance by considering the following two scenarios:
-
When detector evidence is strong but predictor evidence is weak: The predictor has not experienced substantial performance degradation, but the detector is more likely to raise a drift alarm. This may lead to unnecessary adaptation, which can disturb the existing model and cause unstable predictive performance.
- In this case, the detection threshold should be increased to reduce detector sensitivity.
-
When detector evidence is weak but predictor evidence is strong: The predictor has experienced substantial performance degradation, but the detector is less likely to raise a drift alarm. This may delay necessary adaptation and leave the predictor unable to respond promptly to concept drift.
- In this case, the detection threshold should be decreased to increase detector sensitivity.
These two scenarios illustrate why aligning detector evidence with predictor evidence can help improve predictive performance. By adjusting the detection threshold to better reflect the predictor's adaptation needs, unnecessary adaptations could be reduced, and responses to performance degradation could become more timely.
To formalize this intuition, we define the surrogate-optimal threshold as . Our surrogate objective can be equivalently expressed as a regularized KL divergence (please see the derivation below):
\begin{equation} \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad L_t(\theta) = \tau D_{\mathrm{KL}}(\mathrm{Bern}(r_t)|\mathrm{Bern}(q_t(\theta))) + \frac{\lambda}{2\eta}\theta^2 + C_t, \end{equation}
where is independent of . Therefore, represents the threshold that aligns detector evidence with predictor evidence , while regularization helps maintain stable threshold adjustments.
Our dynamic regret bound shows that STAC can track these time-varying surrogate-optimal thresholds in terms of cumulative surrogate loss. This provides a theoretical foundation for adjusting detector sensitivity according to the predictor's adaptation needs, which may help improve adaptation timing and predictive performance.
Derivation. Let and . Treating and as fixed, the KL divergence between two Bernoulli distributions is:
\begin{equation} \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad D_{\mathrm{KL}}(\mathrm{Bern}(r_t)|\mathrm{Bern}(q_t(\theta)))=\log(1+e^z)-r_tz-H(r_t), \end{equation}
where is the binary entropy, with . Rearranging gives:
\begin{equation} \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad L_t(\theta)=\tau D_{\mathrm{KL}}(\mathrm{Bern}(r_t)|\mathrm{Bern}(q_t(\theta)))+\frac{\lambda}{2\eta}\theta^2+C_t, \end{equation}
where is independent of .
Differentiating yields . Applying gradient descent with step size at , where , gives , which is exactly the threshold update of STAC.
On the reliability of .
The purpose of is to accumulate observed performance changes following previous adaptations, rather than estimate the exact benefit of an individual adaptation. We therefore conducted an additional experiment to examine whether the observed performance changes used to update are associated with the benefits of adaptation relative to no adaptation.
For each adaptation event, we compared two versions of the same model: one with adaptation and one without adaptation. We evaluated both on the same next data chunk and compared their accuracy difference with the observed performance change used to update .
We conducted 720 experiments across eight datasets, six drift detectors, three classifiers, and five random seeds, obtaining 10,755 valid comparisons. Across the pooled adaptation events, the Pearson correlation coefficients were 0.969, 0.923, and 0.386 for GNB, VFDT, and DNN, respectively. These results show a strong overall correlation between the observed performance changes used to update and the benefits of adaptation relative to no adaptation for GNB and VFDT, while the correlation is weaker for DNN.
Together, the theoretical analysis explains how STAC adjusts detector sensitivity according to predictor evidence, while the additional experiment provides empirical evidence for the performance feedback underlying .
Thank You for Your Thoughtful Review! We Will Provide a Detailed Response Soon.
Dear Reviewer 3aNW,
I would like to express my sincere gratitude for your insightful and constructive comments! We greatly appreciate the time and effort you devoted to reviewing our paper, as well as your positive assessment of our work.
We are currently preparing additional experiments and analyses to address your concerns. While we are still consolidating some of the results, we would like to share a brief outline of what we are preparing. We will provide a more detailed, point-by-point response once everything is ready:
In response to Weakness 1 and Question 1 (Delayed and partial labels). We will present additional experiments exploring how STAC can be integrated into an existing stream learning framework with delayed labels.
In response to Weakness 2 and Questions 2–3 (Adaptation feedback and theoretical guarantees). We will provide additional empirical analyses of gt, clarify its role in adaptation decisions, and further explain the relationship between our surrogate dynamic regret guarantee and predictive performance.
In response to Weakness 3 and Question 4 (CIFAR-10-CD evaluation). We will further clarify the rationale behind our training configuration, supported by additional experimental evidence.
In response to Question 5 (Figure 1). We are working on revising Figure 1 to provide a more intuitive illustration of the STAC framework.
This outline may change slightly as we prepare our detailed responses.
Thank you again for your thoughtful review and encouragement. We would be grateful for your patience while we prepare our detailed response!