- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies domain generalization classification under sampling pattern shifts in irregularly sampled time series. The authors focus on modern irregular time series models. First, the authors developed a benchmark dedicated to sampling pattern shifts in Irregularly sampled multivariate time series (ISMTS). The benchmark simulates different designed sampling shifts and it was used to test cross shifting patterns and domains. Based on the findings the authors proposed a new method PRISM, that aims to address the limitations of existing work.
The key contributions are the proposed benchmark, the analysis, and the proposed method.
Critical strengths
Pros:
- The analysis is interesting. Some insights (e.g., Fraction of shifted test samples and Shortcut strength analysis in figure 2) can be valuable for the research community.
- The proposed method shows good performance and also under ablation study. Table 1 shows the proposed method is quite strong under the synthetic environment. Tables 2 and 3 also shows the effectiveness of the method.
Critical weaknesses
Cons: The analysis seems to be less supported by the claims and for the model design. Below are some specific feedbacks.
- For the analysis:
- The analysis (sec. 2) in general is a bit hard to understand before the introduction of the task in Sec. 3.1.
- The section of ‘Real-world sampling processes are predictive, yet domain dependent’ can be improved.
- In Figure 4, the authors compare results of ‘observation masks alone’ and ‘full input’. It would be helpful to explain what is the full input. The claim that ‘mask-only prediction markedly under cross-region transfer’ also seems less supported.
- For line 185 “sampling patterns can be both predictive and domain dependent—the two ingredients that make sampling-specific shortcuts brittle under domain shift”, the authors use the analysis of Figure 4 and 1 to explain the results in figure 3. The connections are not clear and no evidence directly this.
- ‘Label’ is used throughout in sec. 2.2 but there is no definition. According to sec. 3.1, does label mean m and t which encode data irregularity? Or is the class to be predicted?
- For the model design:
- The motivation of sec. 3.2 Complementary representation pretraining can be improved. It is not very clear how the controlled representation learning can help mitigate the shortcut and improve DG. How the learn representation are invariant to sampling patterns (line 238).
- Other things that can be clearer
- The settings of ‘bias-conflicting accuracy’ and the ‘mask-following rate’ for Fig. 4.
- Eq. 1, means the ground truth features. Thus, it would be better to be more specific than stating “by randomly hiding additional observations.” (In line 247)
- Explain what the ‘reference’ mean in “The Regular process serves as a reference condition” In line 148.
- It would also be more specific for the domain generalization task in the introduction, cause there are also other tasks than the classification studied in the work, e.g., for time series forecasting, e.g., generalization across domains or over time.
The anonymized code repository also seems expired.
Questions for authors
Please see Critical Weaknesses*
5 comments
[W4]
The motivation of sec. 3.2 Complementary representation pretraining can be improved. It is not very clear how the controlled representation learning can help mitigate the shortcut and improve DG. How the learn representation are invariant to sampling patterns (line 238).
Thank you for this question. First, PRISM does not aim for invariance to sampling patterns: as stated in line 238, invariance "may discard useful predictive cues", such as the more frequent monitoring of deteriorating patients in eICU (Section 2.2). PRISM's design rationale is:
- Pretraining without task labels. Under task supervision, sampling patterns become shortcuts as simple, directly observable cues (ERM: 94.0±2.4 bias-aligned vs. 42.6±1.9 bias-conflicting accuracy, Table A6), which motivates pretraining rich representations without task labels.
- Separate encoders. Both kinds of information are needed and complementary: E_feat recovers removed feature values (Eq. 1), E_samp reconstructs the observation mask (Eq. 2); a larger single encoder trained with both objectives underperforms PRISM on HAR-C and eICU (7rtk, W1).
- Worst-view objective downstream. Even with separate encoders, the classifier may ignore the feature representation and use E_samp alone, so PRISM minimizes the worst loss over K sampling variations (Section 3.2), making reliance on a cue that some variations disrupt costly (HAR-C: 72.02±0.43 without variations vs. 80.57±0.19, Table 3).
This design directly reduces shortcut reliance: on the shortcut benchmark (ρ=0.9) with the same sampling variations, complementary pretraining raises bias-conflicting accuracy from 38.30±1.30 to 49.11±0.74 and lowers the mask-following rate from 0.498±0.011 to 0.405±0.015 (p-value=1e-5 and 1e-4; 7rtk, W2), while a single encoder with the same variations follows the sampling cue even more (0.556±0.016). Table 3 shows the DG gain from complementary pretraining. We have revised the start of Section 3.1 accordingly.
[W5]
The settings of ‘bias-conflicting accuracy’ and the ‘mask-following rate’ for Fig. 4.
Thank you for pointing this out. We have added the following definitions to Section 2.2, where Figure 3 (varying ρ) and Figure 4 (ρ=0.9) are discussed.
- Setting. Each of the six HAR classes is tied to a distinct sampling pattern, all with the same observation budget; each training sample is observed with its class's pattern with probability ρ (ρ=0.9 in all four source environments for Figure 4) and with one of the other five otherwise.
- Test sets. The same test samples are observed with their class's pattern (bias-aligned), a pattern drawn independently of the label (unbiased), or the pattern of a class y_m other than the true label y (bias-conflicting).
- Metrics. Bias-conflicting accuracy is the accuracy on the bias-conflicting set; the mask-following rate is the fraction of bias-conflicting samples predicted as y_m, i.e., following the sampling cue rather than the true label (chance level: 1/6). Less shortcut reliance means higher bias-conflicting accuracy and a lower mask-following rate.
[W6]
Eq. 1, [...] means the ground truth features. Thus, it would be better to be more specific than stating “by randomly hiding additional observations.” (In line 247)
Thank you for the suggestion. In Eq. 1, v denotes the ground-truth feature values, and the loss covers only entries observed in x but removed by T (m = 1, m̃ = 0). Were entries visible in x̃ included, E_feat could minimize the loss by an identity mapping that copies its input, capturing little beyond raw values; recovering removed values instead requires encoding how values evolve over time and relate across features, yielding a richer representation. By "randomly hiding additional observations" (line 247), we meant hiding some observed entries of x (m = 1) beyond those already unobserved (m = 0); as this was unclear, the revised sentence states that T removes the observations of x at randomly selected time points and features, with the drop ratios at the end of Section 3.1 (Table A5).
[W7]
Explain what the ‘reference’ mean in “The Regular process serves as a reference condition” In line 148.
By "reference", we meant that performance is maintained under regular sampling, unlike under the other shifts (ERM: 80.97±0.81% vs. 79.92±0.55% in the source domain (Random), Table 1; 54–69% under the other shifts). Regular sampling observes all features at evenly spaced time points, differing from Random only in this spacing: in both, all features are observed together and observations span the whole sequence. As the term was unclear, we have replaced it with this explanation (Sections 2.1 and 2.2).
[W8]
It would also be more specific for the domain generalization task in the introduction, cause there are also other tasks than the classification studied in the work, e.g., for time series forecasting, e.g., generalization across domains or over time.
To make our scope explicit, we have revised the introduction (Section 1) and the problem setup (now at the beginning of Section 2) to state that we study domain generalization for classification of irregularly sampled multivariate time series.
[W9]
The anonymized code repository also seems expired.
Thank you for pointing this out. The anonymous repository has been restored with the code for all new experiments (https://anonymous.4open.science/r/PRISM-8C66), and the link is updated in the OpenReview abstract and the revised paper (Abstract, Section 4.1, Reproducibility Statement).
We sincerely thank the reviewer for the thoughtful, detailed feedback; addressing the comments on the writing has substantially improved the clarity and quality of our paper. We especially appreciate the positive evaluation of (1) the interesting analysis, with insights such as the fraction of shifted test samples and the shortcut strength analysis (Figure 3) being valuable to the community, and (2) the strong performance of our method, including in the ablation study (Tables 1–3). We address all concerns below.
[W1]
We thank the reviewer for these suggestions. We have moved the problem setup (formerly Section 3.1) to the start of Section 2, so the task and all symbols, now used in Section 2, are defined before the analysis (former Sections 3.2 and 3.3 are now 3.1 and 3.2). "Label" denotes the class label y of the downstream task (e.g., in-hospital mortality on eICU, the activity on HAR-C), never the sampling information m or t, as the problem setup now states.
[W2]
Thank you for these suggestions.
Full input is the model input used throughout the paper: observed values, observation mask and timestamps. In "mask only", the observed values are replaced by the observation mask, so the model sees where and when, but not what, was measured. We have added this definition to Section 2.2.
Predictive, yet unstable across regions. With 50% AUROC as random guessing, observation masks are clearly predictive, exceeding chance by 15.3±0.3 points in-domain on average, but lose 5.8±0.5 points in the worst other region (Figure 4, left), about a third of this margin. On the same test set, too, a model trained in another region is worse than the in-region model in 11 of 12 region pairs (AUROC, 5 seeds; columns are target regions):
We agree the original text did not quantify this drop and have added these numbers to Section 2.2.
[W3]
Thank you for pointing this out. We understand this reading: the eICU paragraph announced "the same phenomenon" and called its findings "the two ingredients that make sampling-specific shortcuts brittle", as if they explained Figure 3. However, Figure 3 is a self-contained controlled HAR-C experiment that directly shows this brittleness; Figures 1 and 4 (left) only check whether its conditions also arise in real data. Our logical flow is:
We have revised line 185 and the opening of the eICU paragraph accordingly.