- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies whether a change detector signal is sufficient to determine an appropriate model adaptation strategy. It introduces unactionability through post-drift distributions that produce the same detector signal but favour different adaptation operators. It also derives an information-theoretic bound relating detector capacity to the probability of misadaptation. Synthetic experiments illustrate the proposed drift scenarios, while a case study across 24 streams from the USP DS repository examines the utility of fixed detector-triggered adaptation across different learning architectures.
Critical strengths
- The paper addresses an important distinction between change detection and model adaptation. Detecting a change does not by itself determine whether to retain, reset, or partially adapt a learner. The finite-horizon risk formulation is well motivated, as it accounts for both the immediate cost of an adaptation operation and the learner’s subsequent recovery.
- The case study provides useful evidence on the effects of fixed adaptation policies. The evaluation covers decision trees, ensembles, kNNs, and neural networks, and includes comparisons with and without change detectors. The results show that detector-triggered adaptation can help some architectures and hurt others. This supports the paper’s argument that adaptation policies should account for the learner and the data stream.
Critical weaknesses
- The proofs of Theorems 2–4 do not establish the claimed finite-horizon risk comparisons. In Section 3.1, the learner continues to update after either the No-Action or Reset operator is applied. However, the proof sketches in Appendices A.2–A.4 assume that retaining the existing learner leads to persistent error, while resetting allows it to learn the new decision boundary. The retained learner can also learn from post-drift samples. Without specifying the online learning rule, reset initialization, and horizon (H), it is unclear why reset should have strictly lower finite-horizon risk. Appendix A.1 also contains a smaller proof error: the same lower bound does not necessarily hold after randomizing over adaptation operators. For example, excess-risk vectors ((0,1)) and ((1,0)) have worst-case risk 1 individually, but their uniform mixture has worst-case risk (1/2).
- The information bottleneck analysis applies to a restricted adaptation policy. Theorem 5 assumes that the adaptation policy is conditioned strictly on the detector signal. In practice, a policy may also use the learner state, previous detector signals, and the time of detection. The one-bit bound therefore applies to a single binary signal, rather than necessarily to all information available for adaptation. Moreover, a non-zero probability of misadaptation does not imply a large excess risk: a non-optimal adaptation operator may perform almost as well as the optimal one. The paper should make this distinction clearer when discussing the practical implications of the bound.
- The empirical results provide limited evidence for unactionability itself. The USP DS case study shows substantial variation in the utility of fixed detector-triggered adaptation. However, this variation could also arise from detection delay, parameter settings, or the cost of resetting. Comparisons between different architectures, such as AMF and ARF, cannot separate these effects. A more direct evaluation would keep the learner, detector, and candidate adaptation operators fixed, and vary the information available to the adaptation policy. The synthetic experiments also use a single post-drift horizon and default learning parameters, leaving unclear how robust the preferred adaptation operation is to these choices.
- The novelty of the general theoretical results is limited. Theorem 1 assumes that no adaptation operator is optimal for every distribution in the post-drift fibre, and concludes that a policy observing only that signal must incur positive excess risk somewhere in the fibre. This is a useful formalization, but the conclusion follows quite directly from the assumption. Theorem 5 similarly applies the data-processing and Fano inequalities to the stated observation model. The main theoretical contribution therefore appears to rest on the drift-specific constructions in Theorems 2–4, whose proofs currently leave the central risk comparisons unresolved.
- Several inconsistencies make the constructions and experiments difficult to follow. Section 3.4 samples post-drift covariates from ([0.5,1]^2), whereas Appendix D.1 changes only (X_1) to (U[0,0.5]). For latent task inversion, Appendix B.2 changes (Z_2), while D.1 changes (Z_1). These settings are not equivalent. The notation (z) is also used for the binary detector signal, the label-noise probability, and a distributional discrepancy. In Appendix A.2, the posterior under symmetric label noise should be (z+(1-2z)\mathbf{1}(X_1+X_2>1)); the additive (z) is missing. These points need correction so that the theoretical constructions can be matched to the reported experiments.
Questions for authors
- Under what assumptions on the online learning rule, reset initialization, and horizon (H) do Theorems 2–4 establish that reset has strictly lower finite-horizon risk when both learners continue updating? Can the argument in Appendix A.1 also be corrected for randomized policies?
- How far do the information bottleneck results extend beyond policies conditioned on a single detector signal? In particular, how would access to learner state and previous detector signals affect the bound, and when does a non-zero probability of misadaptation imply meaningful excess risk?
- Could the authors clarify which synthetic settings generated the figures, given the differences between Section 3.4 and Appendix D.1, and between Appendices B.2 and D.1? How sensitive are the reported advantages of reset or no action to the post-drift horizon and learning parameters?
3 comments
Thank you for the detailed response and revisions. The clarification of the information bound’s scope and the corrections to the synthetic settings help address several of my original concerns. Specifying the learner and horizon conditions, together with the paired retain/reset experiments, also represents meaningful progress. Regarding Weakness 1, resolution of the original soundness concerns depends on the revised proofs establishing the finite-horizon risk comparisons with both learners updating, together with the corrected randomized-policy bound. Regarding Weakness 3, the oracle’s action entropy is informative, but should be interpreted alongside the paired excess-risk results and their uncertainty to distinguish genuine differences in preferred operations from near-ties or sampling variability. Regarding Weakness 4, the strengthened characterization clarifies the formalization, but the substantive theoretical contribution still rests on the validity and informativeness of the drift-specific constructions under the stated assumptions.
Response to Comment by Reviewer ZTiY
Thank you for engaging with the revision.
W1/W4: Validity of the drift-specific constructions:
To validate the finite horizon comparisons, we introduced two lemmas (Lemmas 1–2, App. A.2):
Lemma 1 (stability). We couple the two learners on the same post-drift samples. In every cell, the retained learner’s class-count margin is always at least the reset learner’s margin plus . When the Bayes classifier is unchanged, expected error is non-increasing in this margin, so retaining is never worse at any step. It is strictly better on every first visit to a cell, which gives for all .
Lemma 2 (inertia). Under label flips, the retained learner makes mistakes on the first visits to each flipped cell, whereas the reset learner only makes mistakes on first visits (expected loss = 0.5). Eq. 6 gives the exact difference in risk. The horizon condition follows from a Chernoff bound on the number of visits.
Theorems 6–8 apply these two lemmas to the three drift settings, and Theorem 3 summarises them. The randomised-policy bound (App. A.1) follows by averaging: for any mixture , .
W3: Action entropy vs. near-ties on USP DS:
In the real-world streams, exploring all action sequences is infeasible, so the oracle is obtained with a beam search (width 20). All learners and detectors are deterministic and streams order is preserved, thus there is no run-to-run randomness. We account for the remaining finite-sample variation in with our statistical test (App. D.2). Table 2 excludes settings where the oracle and both fixed policies do not differ.
For 40/89 settings (15 of the 24 streams, across all four detector–model pairs), the oracle statistically beats both always-retain and always-reset. This is only possible if some detections require retaining while others resetting, even though every detection produces the same signal. This is the condition of Theorem 1 (the unactionability theorem), observed on real streams. In the remaining settings, one fixed action is already near-optimal, which is consistent with our claim that unactionability can occur, not that it always does.
We agree that individual decisions may still be near-ties. We therefore treat action entropy as descriptive, not necessarily as a count of decisions that matter.
W4: Contribution:
We agree that the contribution rests on the drift-specific constructions. We acknowledge that limitation which is why we include formal proofs and empirical evidence of unactionability on three categories of drift (loss / supervised, unsupervised / covariate, and latent representation) which we believe covers the vast majority of drift settings as noted in [1]. Covariate drift is unactionable for every horizon. Posterior and representation drift are unactionable for horizons above an explicit threshold set by the learner’s inertia .
[1] Vitor Cerqueira, Heitor Murilo Gomes, Marco Heyden, Bernhard Pfahringer, and Albert Bifet. A framework for evaluating and benchmarking concept drift detection methods. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pp. 8671–8682, 2026.
Answers for rebuttal
Thank you for your detailed and constructive comments. We agree with each soundness point and have revised accordingly.
W1/Q1. Finite-horizon risk with both learners updating: The original sketches implicitly assumed the retained learner does not adapt. We now give full proofs for a specified learner: a partition count learner, which is the leaf model of a majority-class Hoeffding tree with fixed structure (Def. 3). Reset sets the counts to zero, and both learners update after adaptation. We introduce two new properties: stability and inertia (Lemmas 1–2) to explain these results. Theorem 3 (with Theorems 6–8 in App. A) states the horizons explicitly: covariate drift is unactionable for every H≥1, and posterior and representation drift for H above an explicit threshold set by the inertia n0. Remark 8 notes that the advantage of reset vanishes as H→∞, so the optimal operator also depends on H and the learner’s state (this is also shown in the sensitivity analyses we added (see the next point)). The randomised-policy error in A.1 is fixed (see our reply to mwjw).
W2/Q2. Scope of the information bound: Theorem 4 holds for any finite signal space, so it covers warning/drift detectors (H(Z) ≤ log 3). We now state its scope explicitly (Remark 3): it covers policies conditioned on the detector output, which is how detector-triggered adaptation is implemented in the methods we review. Additional state information and multi-detector architectures are left to future work. On misadaptation vs. excess risk, we agree: E[δH]≥γPe, which says nothing when γ≈0 (Remark 5). This is why our experiments report each adaptation operation’s excess risk directly.
W3/Q3. More direct empirical evidence: As you suggested, the learner, detector and candidate operators are now fixed. In the synthetic experiments, each replication forks the learner at the first detection and evaluates retain and reset on the same samples, giving paired risks (Sect. 3.4, Table 3). App. B.3 varies the horizon, ADWIN’s threshold, the learning rate and the number of pre-drift samples and show under which conditions the optimal adaptation are still preferred. Sect. 5 applies the same comparison to the USP DS streams for two learners and two detectors, against a hindsight oracle. The evidence for unactionability is that the oracle mixes retain and reset across events that all produce the same signal (action entropy 0.61–0.73), not that it outperforms fixed policies.
W4. Novelty of Theorem 1: We agree it is primarily a formalisation. The revision strengthens it to a characterisation (an if-and-only-if: a zero-excess-risk policy exists exactly when the fibre has a common optimal operator) and a quantitative bound, and the main technical contribution is now the drift-specific results with explicit horizons (Theorem 3, Lemmas 1–2).
W5. We have corrected the inconsistencies you identified. For example, Section 3.4 now uses the non-degenerate covariate shift X1∼U[0,0.5] matching App. D.1. The representation constructions in the theorem and the experiment match. z now denotes only the detector signal, the noise rate is η and the posterior is now η+(1−2η)⋅1[⋅].
Please let us know if you have any additional questions.