- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes an instrumental variable (IV)-based framework for detecting concept drift in data streams. The authors introduce IVDD, which monitors changes in estimated causal effects to detect drift, and IVAT, which incorporates IV-based monitoring into adaptive decision trees. Experiments on synthetic and real-world datasets show competitive performance, particularly when ground-truth labels are delayed.
Critical strengths
The paper is well-written and easy to follow. The problem is real and significant.
Critical weaknesses
-
The validity of instrumental variables is not well justified.
-
The connection between changes in causal effect and concept drift is unclear.
Questions for authors
My questions are shown below:
-
How do the authors ensure that the instrumental variables discovered from real-world datasets are valid, especially when the causal graphs are estimated using LPCMCI?
-
How are hidden confounders and uncertain causal relationships handled?
-
VDD is presented as a label-free concept drift detector. However, when the monitored outcome is the prediction target, causal effect estimation requires labels. When only feature-level relationships are monitored, how do the authors justify that changes in these relationships indicate concept drift in the downstream prediction task?
3 comments
Response to Reviewer 4LFD (Question 3)
3. Why Feature-Level Mechanism Shifts Indicate Downstream Concept Drift (Weakness 2 & Question 3)
Under the Sparse Mechanism Shift hypothesis (Schölkopf et al., 2021), distributional changes typically arise from shifts in a subset of structural mechanisms. Changes in feature-level mechanisms do not necessarily imply changes in the target mechanism; their relevance depends on their causal relationship to the target.
To empirically investigate whether feature-level monitoring provides useful information for adaptation, in the revised manuscript (Appendix E.4, Tables 13–15) we included an ablation under immediate feedback and delayed-label settings (), comparing three IVDD configurations: all discovered IV triplets, target-only triplets (with as outcome), and feature-only triplets (with input features as outcomes, requiring no labels):
Table: IVDD covariate and target triplets ablation: average accuracy per triplet configuration (Table 15 in the revised manuscript).
| Configuration | Immediate | |||
|---|---|---|---|---|
| Target-Only Triplets | 0.740 | 0.734 | 0.723 | 0.711 |
| Feature-Only Triplets | 0.735 | 0.727 | 0.720 | 0.706 |
| All Triplets | 0.738 | 0.729 | 0.720 | 0.711 |
The results show that feature-only monitoring achieves downstream accuracy within – percentage points of target-only and all-triplets monitoring across all evaluated delays.
Crucially, this mechanism-aware feature-level tracking distinguishes IVDD from both traditional supervised baselines and standard unsupervised detectors. While supervised detectors must wait for shifts to degrade downstream accuracy, and unsupervised detectors (like MMDEW or DD-SCC) monitor global feature distributions, IVDD continuously monitors the localized causal structure of the feature space itself. On complex spatial streams like Covertype (Table 14), the Feature-Only and All Triplets configurations outperformed Target-Only across all delay levels (reaching and accuracy at , respectively, compared to for target-only). This confirms that structural feature-level monitoring is not merely a proxy for target drift, but a distinct and advantageous adaptation signal.
References
Joshua D Angrist, Guido W Imbens, and Donald B Rubin. Identification of causal effects using instrumental variables. Journal of the American statistical Association, 91(434):444–455, 1996.
Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669–688, 1995. ISSN 00063444, 14643510.
Bernhard Sch¨olkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612–634, 2021.
Thanks for your answer. I will maintain my score.
Response to Reviewer 4LFD (Questions 1 and 2)
We thank the reviewer for the thoughtful review, for recognizing the significance of the problem, and for evaluating our presentation, informativeness, and soundness positively. We address each point raised by the reviewer directly below.
Estimating causal graphs from observational time series where latent confounding may be present is a classic challenge. We employ LPCMCI (Gerhardus & Runge, 2020) precisely because it does not assume causal sufficiency (i.e., it explicitly allows for unobserved confounders) and returns a Partial Ancestral Graph (PAG). In the revised manuscript, we expanded Appendix D.1 to describe the complete procedure through which our framework extracts candidate instrumental variable triplets (Z,W,T) from the PAG returned by LPCMCI:
Importantly, these criteria constitute a data-driven screening procedure rather than a formal guarantee of instrument validity. Causal discovery from observational data remains subject to its underlying assumptions, and the discovered PAG may retain uncertainty regarding some causal relationships. We have clarified these limitations in the conclusion.
However, despite these theoretical limitations, our framework proves its practical worth empirically. In practice, no causal methodology can completely avoid structural assumptions; yet our extensive real-world benchmarking demonstrates that our data-driven screening successfully bridges the gap between causal theory and online application. The resulting performance gains—especially under delayed feedback—confirm that IV-based monitoring provides tangible, real-world value for adaptive stream learning.
Resolving unobserved confounding is the primary motivation for adopting IVs in our work. Standard regression or conditioning fails to identify causal relationships when an unobserved confounder U acts on both W and T (Pearl, 1995). Under the IV framework, the instrument Z introduces variation into W that is unconfounded by U (Z⊥U). By taking the Wald estimator:
β^=Cov(Z,W)Cov(Z,T),
the influence of the unobserved confounder U cancels out asymptotically, yielding a consistent estimate of the causal mechanism W→T despite the presence of hidden confounding (which we also empirically verify in our new confounder-drift experiment in Appendix E.1, Figure 8).
Uncertain Edges in PAGs: LPCMCI represents uncertainty in causal relationships through PAGs, where circle endpoints (∘) indicate that the corresponding edge orientation cannot be uniquely determined from the observational data alone. Rather than arbitrarily orienting these edges, our extraction rules explicitly account for endpoint marks: accepting Z∘→W only as a candidate link subject to online instrument-strength filtering (τIV=0.10 and IVAT's
INCONCLUSIVEstate in Algorithm 2), while strictly rejecting any adjacency between Z and T regardless of endpoint marks. We have detailed these extraction rules and their implications in Appendix D.1 and in the conclusion.