- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a three-stage, label-free pipeline for detecting, localizing, and attributing covariate shift between two unlabelled datasets. It consists of a pointwise backbone (based on the EagleEye method), a tail-based equalization step, and a Neighbor-Enriched Subspace Localization method that learns non-negative per-feature attribution weights. Validation has two parts: a 20-dimensional Gaussian-mixture benchmark with known ground truth, run across multiple seeds with parameter-sensitivity grids, as well as an application to PTB-XL+ healthy ECG recordings (782 features), where age/sex-matched cohorts are constructed to differ in measurement-device composition. The method prunes 3–6% of each cohort, the pruned sets are enriched in the imbalanced device components, while selected supports span 120–140 of 782 features.
Critical strengths
-
The controlled benchmark demonstrates the capability of the model to detect localized shifts. The model is able to recover a large percentage of the injected samples in most experiments.
-
The ECG example is a well-designed real-world experiment for this problem.
Critical weaknesses
-
The pointwise anomaly scores are calibrated using B(i, K), which depends on a sample's neighbors, which is OK. But then, the tail scores are treated as an i.i.d. sample in a two-sample KS comparison, whereas the scores of nearby points are computed from largely the same pooled neighbours and are strongly dependent. Moreover, Algorithm 1 performs repeated tests without adjusting the significance level to control for the error introduced by multiple testing. As a result, the calibration of the stopping rule needs additional theoretical and empirical validation.
-
The comparative claims are not supported by the required experiments. The introduction motivates the work against MMD, energy distance, C2ST and nearest-neighbor two-sample tests, however none of them is evaluated quantitatively. Furthermore, there is no detection experiment (e.g., power/false-positive at matched level) for any method, including the proposed. And the attribution stage is never compared against any alternative, even though the true support is known exactly on the benchmark. Ablation studies measuring the contribution of each component (equalization alone vs plus NESL) are also not evaluated.
-
Positioning compared to the relevant literature is incomplete. There exist approaches in the literature that deal with the same problem. For example, Kulinski et al., "Feature Shift Detection: Localizing Which Features Have Shifted via Conditional Distribution Tests", NeurIPS 2020; Barrabés et al., "Adversarial Learning for Feature Shift Detection and Correction", NeurIPS 2023; and others. These methods are not discussed or compared in the paper.
Questions for authors
-
How were baseline hyperparameters determined? And why does it make sense to keep them constant? For example, does the MLP training converge after only 300 training iterations?
-
How does the proposed method relate other methods in the literature?