acceptodds
Under review as a conference paper at ICLR 2027

Detecting Concept Drift in Data Streams Using Instrumental Variables

Abstract

Concept drift is a common phenomenon in real-world data streams that requires models to adapt to changing data distributions. Yet, most concept drift detectors raise flags according to changes in observable distributions or predictive error rates, often monitoring proxy signals, and requiring continuous access to ground-truth labels. In addition, they do not provide insight into why or how the data-generating process changed. Although recent work frames concept drift as changes in underlying causal mechanisms, identifying these mechanisms shifts from observational data is challenging due to unobserved confounding. In this work, we track mechanism changes in continuous data streams using Instrumental Variables (IVs). Specifically, we introduce an online IV monitoring framework that assumes an initial causal graph to identify valid instruments and continuously estimates their associated causal effects using the Wald estimator. We demonstrate two complementary applications of this framework. First, the Instrumental Variable-monitored Drift Detector (IVDD) compares successive IV estimates to detect and localize changes in monitored causal relationships without ground-truth labels. Second, the Instrumental Variable-monitored Adaptive Tree (IVAT) incorporates local IV-based causal-effect estimates into online tree learning, using causal-effect heterogeneity to identify relevant subpopulations and local monitoring to guide targeted adaptation. Empirical evaluations on synthetic structural causal models and real-world streams demonstrate that our IV-monitoring approaches pinpoint changes in causal mechanisms, achieving competitive performance against state-of-the-art drift detectors in delayed-feedback scenarios. Source code is available on GitHub.

Primary area:
learning on time series and dynamical systems
Keywords:
data streams, concept drift, instrumental variables, causality

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4LFDOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an instrumental variable (IV)-based framework for detecting concept drift in data streams. The authors introduce IVDD, which monitors changes in estimated causal effects to detect drift, and IVAT, which incorporates IV-based monitoring into adaptive decision trees. Experiments on synthetic and real-world datasets show competitive performance, particularly when ground-truth labels are delayed.

Critical strengths

The paper is well-written and easy to follow. The problem is real and significant.

Critical weaknesses
  1. The validity of instrumental variables is not well justified.

  2. The connection between changes in causal effect and concept drift is unclear.

Questions for authors

My questions are shown below:

  1. How do the authors ensure that the instrumental variables discovered from real-world datasets are valid, especially when the causal graphs are estimated using LPCMCI?

  2. How are hidden confounders and uncertain causal relationships handled?

  3. VDD is presented as a label-free concept drift detector. However, when the monitored outcome is the prediction target, causal effect estimation requires labels. When only feature-level relationships are monitored, how do the authors justify that changes in these relationships indicate concept drift in the downstream prediction task?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4LFD (Questions 1 and 2)

We thank the reviewer for the thoughtful review, for recognizing the significance of the problem, and for evaluating our presentation, informativeness, and soundness positively. We address each point raised by the reviewer directly below.

1. IV Validity and Graph Estimation via LPCMCI (Weakness 1 & Question 1)

Estimating causal graphs from observational time series where latent confounding may be present is a classic challenge. We employ LPCMCI (Gerhardus & Runge, 2020) precisely because it does not assume causal sufficiency (i.e., it explicitly allows for unobserved confounders) and returns a Partial Ancestral Graph (PAG). In the revised manuscript, we expanded Appendix D.1 to describe the complete procedure through which our framework extracts candidate instrumental variable triplets (Z,W,T)(Z, W, T) from the PAG returned by LPCMCI:

  • Treatment–outcome relationship (W,TW, T): A pair (W,T)(W, T) is considered a candidate treatment–outcome pair if the PAG contains W→TW \rightarrow T, W∘→TW \circ\rightarrow T, or W↔TW \leftrightarrow T.
  • Instrument–treatment relationship (Z,WZ, W): We require an arrowhead at the treatment variable WW in the PAG edge connecting ZZ and WW, accepting both fully oriented (Z→WZ \rightarrow W) and partially oriented (Z∘→WZ \circ\rightarrow W) relationships. In PAG semantics, an arrowhead at WW excludes configurations where WW is an ancestor of ZZ (ruling out reverse causation). Because Z∘→WZ \circ\rightarrow W retains uncertainty regarding the instrument-side endpoint, it is treated as a candidate relationship rather than definitive evidence of instrument validity.
  • Instrument–outcome relationship (Z,TZ, T): We conservatively reject candidate triplets whenever any adjacency exists between ZZ and TT in the PAG, regardless of endpoint marks, respecting the exclusion and unconfoundedness assumptions for IVs (Angrist et al., 1996).
  • Statistical safeguards: During streaming execution, IVDD and IVAT evaluate instrument strength (∣Corr(Z,W)∣≥τIV=0.10|\text{Corr}(Z, W)| \ge \tau_{\text{IV}} = 0.10) to mitigate weak-instrument instability and apply Bonferroni correction across simultaneously monitored structures to control multiple testing (Section 4.3).

Importantly, these criteria constitute a data-driven screening procedure rather than a formal guarantee of instrument validity. Causal discovery from observational data remains subject to its underlying assumptions, and the discovered PAG may retain uncertainty regarding some causal relationships. We have clarified these limitations in the conclusion.

However, despite these theoretical limitations, our framework proves its practical worth empirically. In practice, no causal methodology can completely avoid structural assumptions; yet our extensive real-world benchmarking demonstrates that our data-driven screening successfully bridges the gap between causal theory and online application. The resulting performance gains—especially under delayed feedback—confirm that IV-based monitoring provides tangible, real-world value for adaptive stream learning.

2. Handling Hidden Confounders and Uncertain Causal Relationships (Question 2)

Resolving unobserved confounding is the primary motivation for adopting IVs in our work. Standard regression or conditioning fails to identify causal relationships when an unobserved confounder UU acts on both WW and TT (Pearl, 1995). Under the IV framework, the instrument ZZ introduces variation into WW that is unconfounded by UU (Z⊥UZ \perp U). By taking the Wald estimator:

β^=Cov(Z,T)Cov(Z,W),\hat{\beta} = \frac{\text{Cov}(Z, T)}{\text{Cov}(Z, W)},

the influence of the unobserved confounder UU cancels out asymptotically, yielding a consistent estimate of the causal mechanism W→TW \rightarrow T despite the presence of hidden confounding (which we also empirically verify in our new confounder-drift experiment in Appendix E.1, Figure 8).

Uncertain Edges in PAGs: LPCMCI represents uncertainty in causal relationships through PAGs, where circle endpoints (∘\circ) indicate that the corresponding edge orientation cannot be uniquely determined from the observational data alone. Rather than arbitrarily orienting these edges, our extraction rules explicitly account for endpoint marks: accepting Z∘→WZ \circ\rightarrow W only as a candidate link subject to online instrument-strength filtering (τIV=0.10\tau_{\text{IV}} = 0.10 and IVAT's INCONCLUSIVE state in Algorithm 2), while strictly rejecting any adjacency between ZZ and TT regardless of endpoint marks. We have detailed these extraction rules and their implications in Appendix D.1 and in the conclusion.

AuthorsOpenReview ↗

Response to Reviewer 4LFD (Question 3)

3. Why Feature-Level Mechanism Shifts Indicate Downstream Concept Drift (Weakness 2 & Question 3)

Under the Sparse Mechanism Shift hypothesis (Schölkopf et al., 2021), distributional changes typically arise from shifts in a subset of structural mechanisms. Changes in feature-level mechanisms do not necessarily imply changes in the target mechanism; their relevance depends on their causal relationship to the target.

To empirically investigate whether feature-level monitoring provides useful information for adaptation, in the revised manuscript (Appendix E.4, Tables 13–15) we included an ablation under immediate feedback and delayed-label settings (L∈{100,500,1000}L \in \{100, 500, 1000\}), comparing three IVDD configurations: all discovered IV triplets, target-only triplets (with YY as outcome), and feature-only triplets (with input features as outcomes, requiring no labels):

Table: IVDD covariate and target triplets ablation: average accuracy per triplet configuration (Table 15 in the revised manuscript).

ConfigurationImmediateL=100L=100L=500L=500L=1000L=1000
Target-Only Triplets0.7400.7340.7230.711
Feature-Only Triplets0.7350.7270.7200.706
All Triplets0.7380.7290.7200.711

The results show that feature-only monitoring achieves downstream accuracy within 0.30.3–0.70.7 percentage points of target-only and all-triplets monitoring across all evaluated delays.

Crucially, this mechanism-aware feature-level tracking distinguishes IVDD from both traditional supervised baselines and standard unsupervised detectors. While supervised detectors must wait for shifts to degrade downstream accuracy, and unsupervised detectors (like MMDEW or DD-SCC) monitor global feature distributions, IVDD continuously monitors the localized causal structure of the feature space itself. On complex spatial streams like Covertype (Table 14), the Feature-Only and All Triplets configurations outperformed Target-Only across all delay levels (reaching 76.9%76.9\% and 78.0%78.0\% accuracy at L=1,000L=1,000, respectively, compared to 75.5%75.5\% for target-only). This confirms that structural feature-level monitoring is not merely a proxy for target drift, but a distinct and advantageous adaptation signal.

References

Joshua D Angrist, Guido W Imbens, and Donald B Rubin. Identification of causal effects using instrumental variables. Journal of the American statistical Association, 91(434):444–455, 1996.

Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669–688, 1995. ISSN 00063444, 14643510.

Bernhard Sch¨olkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612–634, 2021.

Thanks for your answer. I will maintain my score.

Reviewer_xsZ4OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses concept drift in data streams. However, instead of monitoring a classifier’s error rate, or directly comparing the observed distribution across windows, it follows the causal view of drift in Barboza et al. (2026b) and monitors the causal relationships between variables. Its main contribution is that, instead of just using the causal structure over the observed variables, it proposes the use of instrumental variables (Angrist et al., 1996) so that each causal effect can be estimated even when there are unobserved confounders. Under this framing, the paper proposes a drift detector called IVDD that raises an alarm when one of the causal relationship it tracks changes. The paper then proposes an online decision tree that builds on the estimates used in IVDD. The paper then empirically shows that IVDD is capable of detecting drift with different drift (transition) types, and that IVDD provides an advantage over popular drift detection methods when there is a large lag in obtaining the true labels of a data stream. Finally, the paper shows this advantage on delayed labels translates to IVAT having the largest Cohen's kappa when compared to other adaptive tree methods

Critical strengths

The main strength of the paper is that it advances the use of causal structure and relationships for concept drift detection which is an interesting and promising avenue of research. Furthermore, by combining the use of a causal structure between variables and instrumental variables, the proposed drift detection method IVDD is, in theory, able to distinguish and detect drift caused by an actual change in the causal relationship between two variables, and ignores drift from other sources such as an unobserved confounder acting on both variables. However, this only holds under the IV assumptions and that the effect of one variable on the other is the same for every unit, as in the paper’s linear models (lines 349–351). That said, I believe, drift caused by unobserved confounders are a common problem with real-world data and the paper is a promising step to handling this problem.

Critical weaknesses

One of the main motivations of the paper is to identify concept drift due to changes in the causal mechanism as opposed to drift caused by covariate shift or confounding variables (lines 047-056), and the use of instrumental variables is motivated by unobserved confounding (lines 62–70 and 108–110). However, there does not seem to be an a figure or table in the paper that explicitly shows how IVDD or IVAT performs on drift not caused by changing causal relationships. Appendix E.2 does use synthetic data with both exogenous and confounder drifts, however, Figures 6-9 seems to pool them with endogenous and target drifts, so they don't show how IVDD behaves on exogenous and confounder drifts. It would therefore help for the paper to include the same results/figures but separated by drift type and for one of these figures to be included in the main text to test the paper's initial motivation directly and have the overall story of the paper to be more coherent.

Questions for authors
  • line 62: What are the suitable assumptions alluded to here? Is it just the IV assumptions?
  • Line 198-201: Are there any other assumptions (other than the IV assumptions for a valid IV) that needs to hold in order to identify the causal effect of W on T (under an unobserved confounding variable) from observational data?
  • Eq 7: how is λ\lambda chosen?
  • For the real data set in the experiment for RQ2, how many alarms did IVDD sound? How many alarms did the other methods sound? Basically is IVDD's advantage here due to the changes it detects or from raising fewer alarms?
  • What settings did the experiments use for IVDD's tests? For instance, what value did the experiments use for α\alpha?

Minor comments

  • Section 4.4: It would help if paper could expand upon the intuition of the splitting method.
  • Line 303: The value τIV\tau_{\text{IV}} seems be have not been defined.
  • Line 478: gros -> grows
  • Line 377: Briefly explain Cohen's kappa is in the main text
  • Line 226: State that Eqs 1-2 are the Wald estimator since later sections refer to them as that.
3 comments
AuthorsOpenReview ↗

Response to Reviewer xsZ4 (Part 1)

We sincerely thank the reviewer for their positive assessment, clear understanding of our contributions, and constructive suggestions. We are especially grateful for the recommendation to stratify the CaDrift benchmark by drift type, which significantly sharpens the narrative of our paper. Below is our point-by-point response and a summary of revisions made to the manuscript.

Response to Weakness: Performance on Drift Not Caused by Causal Mechanism Changes

The reviewer raises an essential point: since our primary motivation is to distinguish mechanism shifts from unobserved confounding or covariate shifts, pooling all drift types in the synthetic benchmark obscured how IVDD behaves under non-mechanism changes.

In the revised manuscript, we addressed this in two ways:

  1. Controlled Confounder-Shift Telemetry (Appendix E.1, Figure 8): We simulated the 4-IV localization structure under abrupt, gradual, and incremental interventions directly on the latent confounders (U1,U2,U3U_1, U_2, U_3) that shift E[Y]\mathbb{E}[Y] by >7>7 units while holding the true causal mechanisms strictly invariant (βi=2.0\beta_i = 2.0). As shown in Figure 8, IVDD's Wald ZZ-statistics remain flat within the thresholds and trigger zero false alarms, confirming its ability to ignore pure confounder distribution shifts.

  2. Stratified CaDrift Benchmark (Appendix E.3, Figure 13): We stratified detection performance across the four distinct intervention types in the 120-stream CaDrift benchmark. When assessing metrics per drift type, supervised detectors (DDM, RDDM, ADWIN, OPTWIN) achieve higher TPR only when drift events directly affect the target variable, whereas IVDD and MMDEW maintain higher TPR and F1F_1-scores under non-target structural interventions while IVDD keeps false alarms near zero.

Suitable assumptions for identifying causal effects (Q1 and Q2)

Beyond the three core graphical IV assumptions (relevance, exclusion, and unconfoundedness), the "suitable assumptions" required to identify a point causal effect and track mechanism shifts through the Wald estimator with IVDD are:

  1. Stability of Instrument Validity: The instrument must remain valid across distributional shifts. In particular, changes must not introduce a direct causal effect of ZZ on TT (violating the exclusion restriction) or induce dependence between ZZ and unobserved confounders UU. Otherwise, changes in the Wald estimator may reflect violations of instrument validity rather than changes in the mechanism W→TW \rightarrow T.
  2. Non-degenerate First Stage: The instrument must retain sufficient relevance, i.e., Cov(Z,W)≠0\text{Cov}(Z, W) \neq 0. Weak instruments can lead to substantial estimator instability, potentially producing apparent changes unrelated to the underlying causal mechanism. IVDD and IVAT therefore incorporate an instrument-strength threshold (τIV=0.10\tau_{\text{IV}} = 0.10) to guard against such cases.

We have added a dedicated paragraph in Section 4.3 (Assumptions required to identify causal effects) stating these conditions explicitly, which we reproduce here:

"In addition to the standard IV assumptions, IVDD also requires the following conditions to ensure that changes in the empirical Wald estimator reflect shifts in underlying causal mechanisms: 1) stability of instrument validity: the instrument must remain valid across drifts, i.e., changes must not introduce a direct causal effect of ZZ on TT (violating the exclusion restriction) or induce dependence between ZZ and unobserved confounders UU; 2) Non-degenerate first stage: the instrument must retain sufficient relevance, i.e., Cov(Z,W)≠0\text{Cov}(Z,W)\neq 0. Weak instruments can lead to substantial estimator instability, potentially producing apparent changes unrelated to the underlying causal mechanism."

AuthorsOpenReview ↗

Response to Reviewer xsZ4 (Part 2)

Q3: Choice of λ\lambda

In our main experiments, we set λ=0.25\lambda = 0.25, giving mild weight to causal effect heterogeneity across subpopulations while prioritizing discriminative predictive purity. If a valid IV estimate is unavailable at a given node or the instrument is weak, the node defaults strictly to predictive gain (λ=0\lambda = 0). In the revised manuscript, we explicitly state λ=0.25\lambda = 0.25 in Section 5 and include an ablation study on λ∈{0.00,0.25,0.50,0.75,1.00}\lambda \in \{0.00, 0.25, 0.50, 0.75, 1.00\} in Appendix C.3 (Figures 4 and 5).

This reveals that mild causal heterogeneity (λ∈{0.25,0.50}\lambda \in \{0.25, 0.50\}) yields the best accuracy performance, whereas adopting severe causal heterogeneity and ignoring predictive impurity (λ=1.00\lambda = 1.00) degrades it. While the Nemenyi critical difference diagram (Figure 5) ranks λ=0.50\lambda = 0.50 and λ=0.25\lambda = 0.25 as the top two variants with no statistically significant difference between them, we employed λ=0.25\lambda = 0.25 as it achieves the highest absolute average accuracy and promotes predictive purity during node splits while accounting for causal heterogeneity:

Table: Average classification accuracy of IVAT across CaDrift streams for different λ\lambda values (Appendix C.3).

λ\lambda0.000.000.250.250.500.500.750.751.001.00
Accuracy79.72%79.72\%79.88%79.88\%79.85%79.85\%79.51%79.51\%79.36%79.36\%

Q4: Number of alarms sounded on real-world datasets

This is an insightful question. We added in Table 12 (Appendix E.2) the exact number of drift events each detector triggered across all real-world streams and latency policies. IVDD is more conservative than other detectors on average (15.615.6–18.418.4 alarms across latencies), with alarm volume concentrated in streams undergoing substantial shifts. The largest gap occurs on the Covertype dataset, where IVDD triggered 4848 drift events under immediate feedback compared to 670670 for ADWIN and 3,6703,670 for DDM. The only stream where IVDD did not trigger any drift event was Electricity under delayed feedback, which contains only 22 monitorable IV structures.

Crucially, IVDD's strong predictive performance does not come from simply triggering fewer alarms. To verify this, we added a no-drift baseline (a vanilla HT classifier without a drift detector) to Table 2 (and Table 10 in Appendix E.2). In comparison to this no-drift baseline, IVDD achieves higher accuracy across all feedback configurations (0.6660.666 vs. 0.6440.644 at L=0L=0; 0.6390.639 vs. 0.6280.628 at L=100L=100; 0.6160.616 vs. 0.6110.611 at L=500L=500; and 0.6050.605 vs. 0.5950.595 at L=1000L=1000), whereas all competing supervised and unsupervised detectors fall behind the no-drift baseline under delayed feedback (L≥500L \ge 500). This further supports tracking causal effects in data streams, especially when labels are delayed: by measuring cause-effect relationships over time, IVDD signals adaptation when a monitored mechanism changes, rather than when the marginal distribution or delayed error rate fluctuates. Distribution- and error-based detectors frequently fire on benign covariate shifts or delayed error spikes, triggering premature resets that degrade accuracy below a static learner.

Q5: Hyperparameter settings for IVDD In the revised manuscript, we explicitly state our operational hyperparameters in Section 5, Appendix C.2 (Table 5), and Appendix D.1 (Table 8):

  • Significance level α\alpha: Two-sided zz-test with base critical value c0=3.0c_0 = 3.0 (per-test α=2(1−Φ(3.0))≈0.0027\alpha = 2(1 - \Phi(3.0)) \approx 0.0027), dynamically adjusted with a Bonferroni correction over the mm monitored structures, c=Φ−1(1−α/(2m))c = \Phi^{-1}(1 - \alpha/(2m)), and a warning threshold fixed at 0.8c0.8c (2.42.4 when c0=3.0c_0 = 3.0). The hyperparameter c0=3.0c_0 = 3.0 was selected via the ablation in Appendix C.2 (Table 5) and is shared across all datasets.
  • Window size: Sliding window ∣wt∣=500|w_t| = 500, also ablated in Appendix C.2 (Table 5).
  • Bootstrap variance estimation: B=200B = 200 moving-block bootstrap resamples with block step 1010.
  • Instrument strength threshold: τIV=0.10\tau_{\text{IV}} = 0.10.

Minor Comments

  • Splitting intuition: Added rationale explaining that splits with high GIVG_{\text{IV}} isolate subpopulations that respond differently to the treatment, ensuring local subtrees correspond to homogeneous causal regimes that can be independently adapted upon drift.
  • τIV\tau_{\text{IV}} definition: Explicitly defined τIV=0.10\tau_{\text{IV}} = 0.10 as the minimal instrument relevance threshold below which instruments are deemed weak.
  • typo: Fixed ("gros" →\rightarrow "grows").
  • Cohen's Kappa: Added an explanatory note in Section 5 stating that κ\kappa measures classification agreement beyond chance and is robust against class imbalance.
  • Wald estimator: Explicitly introduced Equations (1) and (2) as the Wald estimator.

Thank you for the response, my concerns have been addressed; I will keep my score.

Reviewer_VsvXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

I this work, the authors introduce IVDD and IVAT, a drift detection and decision tree architecture that utilises instrumental variables to detect changes in the causal structure of a data stream, which they then use to diagnose / detect concept drift.

The authors evaluate both IVDD and IVAT on both synthetic and real data streams. In particular they focus on the delayed label setting where IVDD appears to outperform the state-of-the-art.

Critical strengths
  1. Relevance: The paper tackles a very relevant problem in stream learning. Accurate detection and diagnoses of concept drift is central to developing an understanding of when and how to adapt to changes in the data generating process. The causal perspective is an interesting and newly developing way of doing so.

  2. Unsupervised and Supervised detection: I particularly like the ability of IVDD to identify changes in triplets from covariates and triplets that contain the outcome variable Y. This, in theory, enables detection and adaptation from both covariate and posterior shifts.

  3. Beyond simple test-then-train: I appreciate the authors investigating label delay settings which represent (potentially) more realistic evaluation settings.

Critical weaknesses

Readability and Reproducibility:

The paper is very hard to follow. The authors put of lot of important information in the appendix, and it is scattered. Some information, such as the parameters of their detection method, do not seem to be mentioned at all. I also briefly checked the code, and some of their textual descriptions do not match up with the code.

For example, the text describes the adaptation policy of the supervised detectors as a pure reset mechanism. I wanted to make the comment that modern algorithms such as ARF and SRP train background learners when warning threshold are reached, but it appears from the code that some of the supervised detectors with built-in warning thresholds already do this? I would have expected this to be stated. I would also say that it should be implemented for the other detector types (such as ADWIN which usually use two detectors of different thresholds to signal warnings and resets).

In its current state the paper needs quite a bit of editing. I would remind the authors that a reader should not have to scrounge through the source code and appendices to figure out how their methods were evaluated and implemented.

Reliance on non peer-reviewed work:

A lot of the foundations of this paper are based on two papers that do not seem to have been peer reviewed (Both the Barboza et al. papers). In particular, CaDrift was rejected from ICLR last year, and does not seem to have been accepted at another venue since. I remain open to the idea that CaDrift produces concept drift phenomena that is realistic, but, I cannot say this for certain. I look at Figure 6, and I see that ADWIN and DDM's TPR are close to 0.0. These are detectors with a long and verifiable history in stream learning. I am somewhat doubtful that are just useless (which Figure 6 seems to imply).

Missing Experiments:

The authors did not spend much time evaluating the sensitivity of their method. For example, they mention the window dilemma paper in the introduction but do not evaluate their methods' sensitivity to window sample size. I would have expected these results to have been included in the paper. The authors should elaborate more on how their approaches were parameter tuned.

I am also having trouble reconciling the actual contribution of the causal structure monitoring. It is not clear to me that IVDD's performance when coupled with the Hoeffding Tree isn't simply due to its adaptation policy and dual monitoring strategy (unsupervised and supervised triplets). I would think that an ablation which restricts IVDD to covariate triplets (so purely unsupervised) and triplets containing the outcome Y (so purely supervised) could shed light on this, and would be a valuable contribution to the paper.

Based on these concerns, I currently lean towards a weak-reject but remain open to improving the score if the authors sufficiently address my concerns.

Questions for authors

Questions:

  1. Does changing the background training implementation (i.e., to consider dual thresholds for detectors that don't implement a warning signal) meaningfully change the results?

  2. How did you ensure that the synthetic streams produced by CaDrift meaningfully impacted the statistics that the detectors monitor? For example, if a drift does not meaningfully affect the error rate, neither DDM or ADWIN would detect such as change. Would this then affect the TPR (and other metrics) of these detectors?

  3. How sensitive is this method to changes in window size?

Comments:

  1. I would urge the authors to consider citing work from [1]. It predates both the Barboza et al. preprints that have been cited in this work.
  2. The Window Dilemma Paper should be properly cited.
  3. You should not be marketing your method as an unsupervised detector when you do in fact make use of label information. I would expect IVDD to be called a semi-supervised or a hybrid detector. The same goes for the citation of STUDD, I know the authors of that paper called it an unsupervised detector, but the parent model does at some point need labels for training.

[1] Komnick, D., Lammers, K., Hammer, B., Vaquet, V., & Hinder, F. (2025, September). Causal Explanation of Concept Drift–A Truly Actionable Approach. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 396-412). Cham: Springer Nature Switzerland.

9 comments
AuthorsOpenReview ↗

Response to Reviewer VsvX (Part 1)

We thank the reviewer for the constructive feedback and for being explicitly open to increasing the score once the raised concerns are addressed. We address each point directly below.

W1: Readability and Reproducibility.

We thank the reviewer for identifying inconsistencies between the manuscript and implementation, as well as missing experimental details. We have revisited both the implementation and manuscript to address these concerns.

Adaptation policies and implementation consistency. We corrected the description of the adaptation policies to accurately reflect the implemented warning-level and reset mechanisms in CapyMOA. Under immediate feedback, background learners are trained upon warning signals for detectors with warning thresholds, including IVDD. Under delayed feedback, supervised detectors retain their warning-level policies. The label-independent detectors (DD-SCC, DD-KRC, MMDEW, and IVDD) previously triggered resets only upon drift detection when labels became available, replacing the active model after receiving 100100 labeled samples for training. Following the reviewer's suggestion, we extended the warning-level policy to these detectors as well, including IVDD, and reran the experiments accordingly. This correction consistently improved IVDD's performance under delayed feedback, increasing accuracy from 63.8%63.8\% to 63.9%63.9\% for L=100L=100, from 61.2%61.2\% to 61.6%61.6\% for L=500L=500, and from 60.4%60.4\% to 60.5%60.5\% for L=1,000L=1,000. The updated implementation with the warning level can be found in the supplementary material under run_IVDD_delayed_feedback.py.

These revised experiments preserve IVDD's competitive performance across all evaluated policies, especially under delayed label feedback. We have incorporated the corrected adaptation policies into the experimental protocol in Section 5 and updated the corresponding results throughout the manuscript.

Hyperparameter selection and sensitivity. We additionally conducted an ablation study of IVDD's hyperparameters and selected the resulting configuration for the revised experiments. We have included a full discussion of hyperparameter sensitivity in Appendix C (covering both IVDD in Appendix C.2, Table 5 and IVAT in Appendix C.3), as well as a complete summary of the hyperparameter configurations for IVDD and all baseline detectors in Appendix D.1 (Table 8). We report the values achieved for IVDD in the recommended configuration (window size ∣w∣=500|w|=500, c0=3.0c_0=3.0) below:

Table: Ablation study of IVDD hyperparameters (W,c0W, c_0) averaged across abrupt, gradual, and incremental causal mechanism drift. Results report mean ±\pm standard deviation.

Window (WW)Threshold (c0c_0)Recall (%) ↑\uparrowFalse Alarms ↓\downarrowDelay (samples) ↓\downarrowF1F_1-Score ↑\uparrow
5003.0100.0±0.0100.0 \pm 0.00.3±0.60.3 \pm 0.6383±22383 \pm 220.982±0.0300.982 \pm 0.030

Reproducibility and organization. To address the reviewer's concern regarding the reproducibility and organization of the experimental methodology, we revised the manuscript to make the experimental protocol explicit and accessible. In particular, we moved essential methodological details previously scattered across Appendices D and E into the main text in Section 5. We reproduce the revised passage below:

"Model adaptation follows a background-tree strategy, where a secondary HT is trained on samples observed during the warning period and replaces the current model upon drift confirmation, including under delayed feedback. For IVDD, we discover the DAGs using LPCMCI on the initial warmup samples. Table 2 reports accuracy, Cohen's Kappa (κ\kappa), which measures classification agreement beyond chance and is robust to class imbalance, and average rank across streams based on the accuracy. Further details regarding the experimental setup for the real-world evaluation are also provided in Appendix D.1."

We additionally reorganized Appendices D.1 (including Table 8 with all detector hyperparameters) and E.2 (Tables 10–12), which report the complete experimental results, to improve their readability and facilitate comparisons across methods. More broadly, we improved the self-containedness and accessibility of the manuscript: the revised paper progresses cleanly from the background and formalization of concept drift and structural causal models to the methodological details underpinning our contributions, explicitly articulating our main claims and presenting the corresponding experimental evidence in the main text. The appendices are reserved for supplementary methodological details and extended results, rather than essential information required to understand or validate our claims.

AuthorsOpenReview ↗

Response to Reviewer VsvX (Part 2)

W2: Reliance on non peer-reviewed work. ADWIN and DDM presented low TPR.

We appreciate the reviewer's scrutiny regarding the references used in our work, as well as the recommendation to include the recent contribution by Komnick et al. (2025), which we have incorporated into Sections 1 and 3 of the revised manuscript. We would also like to clarify the respective roles of the theoretical foundations and empirical evaluation in our work. Our proposed framework is grounded in established instrumental-variable theory, which provides the basis for our methodological contributions. In particular, the identification and monitoring of causal mechanisms through IVs—which constitute the core of our approach—rely strictly on classical, peer-reviewed causal inference literature (Pearl, 1995; Angrist et al., 1996; Andrews et al., 2019).

Regarding the synthetic evaluation, we use Barboza et al.'s causal taxonomy to characterize drift according to the intervention applied to the underlying SCM. CaDrift provides a complementary experimental framework for generating controlled data streams with known causal structures and intervention points, allowing us to systematically investigate the behavior of our approach under different types of causal mechanism changes.

Importantly, our primary empirical claims regarding the performance of IVDD and IVAT, particularly their advantages under delayed label feedback, are supported by experiments on real-world streaming benchmarks from the USP DS repository (Tables 2–4). These experiments evaluate our methods under both immediate and delayed feedback protocols and compare them against established, peer-reviewed approaches to concept drift detection and adaptation. Thus, the synthetic experiments provide controlled evidence about specific properties of our approach, while the real-world benchmarks establish their empirical performance in practical streaming settings.

Why ADWIN and DDM present low pooled TPR in CaDrift. We further investigated the behavior of well-established drift detectors (such as ADWIN and DDM) in the experiments with CaDrift, and there is a clear explanation for their low pooled TPR in Figure 9. When stratifying the detectors' performance per intervention type, ADWIN, DDM, and other supervised detectors achieve substantially higher TPR under Target drift (a drift that directly changes the mechanism of the target variable YY). On the other three types of drift (Exogenous, Endogenous, and Confounder), these error-based detectors present low TPR because upstream interventions do not necessarily degrade the immediate classification error rate, which caused their pooled average TPR to plummet. We included this stratified analysis in Appendix E.3 along with Figure 13 illustrating the metrics per drift type.

Q1: Dual-threshold ADWIN

We thank the reviewer for this sharp observation. ADWIN's implementation in CapyMOA indeed does not implement a warning threshold by default. Following the reviewer's feedback, we implemented a dual-threshold ADWIN (δdrift=0.002,δwarn=0.02\delta_{\text{drift}} = 0.002, \delta_{\text{warn}} = 0.02; Table 8) and reran the benchmark experiments. These results improve ADWIN's overall rank in Table 2, though IVDD remains the top-performing detector under delayed feedback (L=500L=500 and L=1000L=1000) and RDDM remains top-ranked under immediate feedback. These results also remain consistent with the low TPR delivered by ADWIN under non-target drift events discussed above.

AuthorsOpenReview ↗

Response to Reviewer VsvX (Part 3)

Q2: How did you ensure that the synthetic streams produced by CaDrift meaningfully impacted the statistics that the detectors monitor?

Indeed, an intervention in an SCM does not necessarily induce a detectable change in every statistic monitored by a drift detector. In particular, interventions affecting exogenous, endogenous, or confounder mechanisms may alter the observed distribution without necessarily changing the target mechanism or prediction error. Consequently, supervised detectors relying on changes in prediction error may not respond to such interventions. This underscores a key advantage of IVDD: by monitoring structural cause-effect relationships directly, our framework is able to identify these causal shifts, enabling adaptation without waiting for error spikes or delayed labels. Complementing our reply above, we stratified detector performance by intervention type (Appendix E.3, Figure 13). The results confirm that ADWIN, DDM, and other supervised detectors achieve substantially higher TPR under target drift than under exogenous, endogenous, and confounder drifts:

Table: Mean True Positive Rate (TPR) stratified across drift mechanisms on the CaDrift synthetic benchmark.

DetectorEndogenousExogenousConfounderTarget
IVDD0.3440.3780.2670.100
MMDEW0.6780.6560.5560.167
ADWIN0.1000.0780.0440.378
DDM0.0780.0560.0330.344

This stratified performance highlights a divergence in monitoring objectives. Supervised methods (e.g., ADWIN, DDM) evaluate the global predictive boundary, primarily triggering under Target drift where interventions immediately degrade predictive performance. In contrast, unsupervised distribution-based methods like MMDEW are highly sensitive to upstream Exogenous (0.6560.656) and Endogenous (0.6780.678) shifts, as they track the global feature space. IVDD targets the structural middle ground: it achieves substantial recall on non-target drifts (0.2670.267–0.3780.378) because it monitors intermediate (Z,W,T)(Z, W, T) triplets distributed throughout the graph. When upstream interventions occur, the propagation of shifts in covariates can change relative causal effects across these localized mechanisms. By continuously monitoring the Wald estimator, IVDD detects these underlying structural shifts as they happen, decoupling adaptation from both immediate target error and purely surface-level distribution changes. While this sensitivity can lead to unnecessary global classifier resets when the target mechanism remains invariant (note that under pure confounder shifts where structural mechanisms remain linear and invariant, IVDD triggers zero false alarms, as shown in Appendix E.1, Figure 8), it becomes highly advantageous when coupled with intervention-aware adaptation policies. Specifically, it provides the precise structural localization required to trigger targeted subtree updates, motivating the localized adaptation strategy employed by IVAT.

Feature-only triplets and target-only triplets

We added an ablation isolating feature-only IV triplets ({Z,W,T}⊆X\{Z, W, T\} \subseteq X) and target-only IV triplets (Y∈{Z,W,T}Y \in \{Z, W, T\}), now detailed in Appendix E.4 (Tables 13–15). We summarize the average accuracy across the streams admitting both triplet types below:

Table IVDD covariate and target triplets ablation: average accuracy per triplet configuration (Table 15 in the revised manuscript).

ConfigurationImmediateL=100L=100L=500L=500L=1000L=1000
Target-Only Triplets0.7400.7340.7230.711
Feature-Only Triplets0.7350.7270.7200.706
All Triplets0.7380.7290.7200.711

The results show that purely unsupervised feature-only monitoring achieves downstream accuracy within 0.30.3–0.70.7 percentage points of target-only and all-triplets monitoring across all evaluated delays. Furthermore, as shown in the per-dataset breakdown in Table 14 (Appendix E.4), the benefit is dataset-dependent: on complex spatial streams like Covertype, Feature-Only and All Triplets outperform Target-Only at every latency (76.9%76.9\% and 78.0%78.0\% vs. 75.5%75.5\% at L=1000L=1000). On NOAA, by contrast, Target-Only performs best (72.8%72.8\% vs. 70.1%70.1\% at L=1000L=1000), since its target monitors never fire and the learner avoids unnecessary global resets. These results also demonstrate that feature-level mechanism monitoring provides an effective, label-free adaptation signal beyond target-error monitoring, while the NOAA result reinforces why localized, intervention-aware adaptation (as in IVAT) is preferable to global model resets when upstream shifts do not alter the target mechanism.

Comments on the paper

1 comment

We want to thank all of the reviewers for their time in reading our paper and providing important and insightful feedback that helped us enhance the quality of our work. All of the reviewers' comments were read carefully and taken into consideration in this rebuttal. Among other points, in the revised manuscript we have: 1) revisited our implementation and rerun the experiments under delayed feedback using the warning level of the label-free detectors; 2) Included ablation studies on IVDD and IVAT hyperparameters (Appendices C.2 and C.3, Table 5), as well as a comprehensive table exposing the hyperparameter configurations of all evaluated detectors (Appendix D.1, Table 8); 3) Added key experimental details that were missing in the first version of the manuscript (Section 5 and Appendix D.1); 4) Explained in detail how the IVs were extracted through the LPCMCI causal discovery algorithm (Appendix D.1); 5) Reassessed the experiments on synthetic data, including a controlled confounder-shift experiment (Appendix E.1, Figure 8) and stratifying the CaDrift analysis across drift types (Appendix E.3, Figure 13); 6) Reported additional results such as the number of drifts detected by each detector (Appendix E.2, Table 12) and a no-drift Hoeffding Tree (HT) baseline (Section 5, Table 2, and Appendix E.2, Table 10); 7) Assessed IVDD on real-world data across different types of IV triplets (feature-only and target-only triplets; Appendix E.4, Tables 13–15).

We also enhanced the writing throughout the paper to further strengthen the narrative and presentation of our work. We hope that our efforts assist in your reassessment of our submission and support raising the assigned scores.

We would also like to extend our thanks to the Area Chair for handling this submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.