- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies the problem of spurious correlations in linear probes, where a classifier may rely on simple but irrelevant features instead of the actual features needed for classification. The main observation is that linear classifiers tend to favor feature directions associated with large eigenvalues of the representation covariance matrix, and these high-variance directions may correspond to spurious features. To address this, the authors apply whitening to the learned representations before training the linear probe. Whitening equalizes the eigenvalues and removes the classifier's preference for high-variance directions. The authors provide theoretical analysis of this behavior and evaluate the approach on synthetic data and several spurious-correlation benchmarks, including Waterbirds, CelebA, and MultiNLI. The experiments show that whitening can substantially improve worst-group accuracy in some settings and can also complement existing approaches for handling spurious correlations.
Critical strengths
The main observation of the paper is interesting: linear probes can favor feature directions associated with large eigenvalues of the covariance matrix, which can increase their reliance on simple or spurious features. Based on this observation, the paper proposes a simple solution using whitening to remove this preference. I also find it interesting that such a relatively simple preprocessing step can provide substantial improvements on some spurious-correlation benchmarks without requiring prior knowledge of the spurious features or additional labels.
Critical weaknesses
My main concern is that whitening changes the geometry of the representation by equalizing the eigenvalues of the covariance matrix. While this can help when directions with large eigenvalues correspond to spurious features, there is no guarantee that this is always the case. High-variance directions may contain useful information for the actual task, and suppressing their relative importance could therefore hurt performance on data where spurious correlations are absent or do not align with these directions. The weaker results on MultiNLI already suggest this limitation. I would like to see a more extensive evaluation on standard datasets without intentionally introduced spurious correlations to better understand when whitening helps and when it can hurt.
Questions for authors
How does whitening affect performance on standard datasets where there are no known or intentionally introduced spurious correlations? Could equalizing the eigenvalues hurt performance by reducing the importance of genuinely useful high-variance features?
Is there a way to determine when large-eigenvalue directions are likely to represent spurious features versus useful features before applying whitening?
The results on MultiNLI suggest that whitening is not always beneficial. Can the authors provide more insight into what properties of a representation or dataset determine whether whitening will help?
Have the authors considered partial or adaptive whitening, rather than forcing all directions toward equal variance, to preserve useful representation geometry while reducing the dominance of unusually large eigenvalues?
3 comments
High-variance directions may contain useful information for the actual task, and suppressing their relative importance could therefore hurt performance on data where spurious correlations are absent or do not align with these directions. The weaker results on MultiNLI already suggest this limitation.
How does whitening affect performance on standard datasets where there are no known or intentionally introduced spurious correlations? Could equalizing the eigenvalues hurt performance by reducing the importance of genuinely useful high-variance features?
I would like to see a more extensive evaluation on standard datasets without intentionally introduced spurious correlations to better understand when whitening helps and when it can hurt.
We agree that this is a possibility, and have added several experiments to the paper to investigate this. First, we have added an experiment where we create a version of Waterbirds where the spurious correlation is absent - e.g. there is no correlation between the background type and bird type. In this experiment, whitening does not lead to a statistically significant reduction in performance.
Second, we have added an experiment on CIFAR-10, where we compare re-training the final layer with no preprocessing, standardization, and whitening, from a finetuned Resnet50. This is a standard benchmark without an intentionally introduced spurious correlations. We again observe no statistically significant reduction in performance from whitening.
Third, we vary the scale of the spurious feature in the synthetic experiment and observe an asymmetry. Whitening substantially improves performance when the spurious feature has a large scale, while having little effect when its scale is equal to or smaller than that of the core feature. This provides further insight into the circumstances under which whitening can help, and why otherwise it does not degrade performance (see the new paragraph near line 275 for details).
These three experiments show no signs that whitening reduces performance in these datasets. While we cannot guarantee that the concern of the reviewer never is an issue, they at least provide empirical evidence that it is a limited risk.
Is there a way to determine when large-eigenvalue directions are likely to represent spurious features versus useful features before applying whitening?
We thank the reviewer for this suggestion. This is also something we considered ourselves. Ideally, one could measure the extent to which spurious features (e.g. backgrounds) align with high eigenvalues, and then compare this to the extent core features (e.g. bird characteristics) align with high eigenvalues. However, we do not deem this viable for two reasons. First, it is very difficult to unambiguously identify separate directions in the feature space associated with the spurious and core feature. While attempts at this have been made (see [7]), these methods do not guarantee directions solely associated with either of the features. On top of that, there is also no guarantee that there exists a unique direction representing either feature. Second, even if we knew how to determine the directions representing spurious and core features, it is impractical to estimate these directions; this likely would require pre-existing knowledge of the spurious correlation as well as the availability of group labels. As such, we believe that when group labels are available for the validation set, it is best to use worst-group accuracy to measure whether whitening improves robustness. Otherwise, class-balanced accuracy is the best alternative, although it can miss performance on (small) subgroups. A careful examination as to how one can select the best preprocessing approach (none, standardization or whitening) beforehand is outside the scope of the paper.
[7] Holstege et al. https://arxiv.org/abs/2310.11991
Have the authors considered partial or adaptive whitening, rather than forcing all directions toward equal variance, to preserve useful representation geometry while reducing the dominance of unusually large eigenvalues?
We agree that partial whitening is an interesting approach, and has the potential to address some limitations of the paper.
Our analysis focuses on full whitening, as this allows us to isolate the effect of removing the max-margin classifier's preference for high-variance directions. A systematic investigation of partial whitening is beyond the scope of this paper, as partial whitening brings its own challenges and complexities - mainly, how one can determine the optimal amount of whitening beforehand. We have added partial whitening as a suggestion for future work in Section 6.
I am okay with acceptance
Considering the response and other reviews. I am okay to accept.
We thank the reviewer for their thoughtful feedback, as well as their suggestions for improvement, which we found very valuable. We also thank them for commenting that it is `interesting that such a relatively simple preprocessing step can provide substantial improvements on some spurious-correlation benchmarks'. Below, we address the feedback point-by-point.
We thank the reviewer for highlighting this limitation of our work. We agree that there is no guarantee that directions associated with large eigenvalues correspond to spurious features. We explicitly acknowledge this limitation in the paper (lines 394–397 and 525–529), but agree this deserves more attention. We have thus updated our introduction (lines 52-55 in the new manuscript) to highlight this.
Our paper establishes that the max-margin classifier has a preference for features associated with large eigenvalues of the covariance matrix, and that whitening removes this preference. This holds regardless of whether these features correspond to spurious or core features. However, whether whitening improves robustness depends on the extent to which these high-variance directions correspond to spurious features. When they contain useful information for classification, we agree that whitening may hurt performance.
As discussed in Sections 2 and 3, a substantial body of previous work relates the reliance of DNNs on spurious correlations to simplicity bias [1–6]. This provides some motivation for expecting spurious features to be associated with high-variance directions, and thus that at least in some cases (as shown in the paper) that whitening can improve robustness to spurious correlations.
[1] Shah et al. (2020) https://proceedings.neurips.cc/paper/2020/hash/6cfe0e6127fa25df2a0ef2ae1067d915-Abstract.html
[2] Pezeshki et al. (2021) https://proceedings.neurips.cc/paper_files/paper/2021/file/0987b8b338d6c90bbedd8631bc499221-Paper.pdf
[3] Teney et al. (2022) https://openaccess.thecvf.com/content/CVPR2022/html/Teney_Evading_the_Simplicity_Bias_Training_a_Diverse_Set_of_Models_CVPR_2022_paper.html
[4] Vasudeva et al. (2024) https://arxiv.org/abs/2310.06161
[5] Yang et al. (2024) https://proceedings.mlr.press/v238/yang24c.html
[6] Qiu et al. (2024) https://proceedings.mlr.press/v235/qiu24a.html
We believe that two properties are particularly relevant.
First, the original classifier should exhibit a substantial preference for simple features, as defined in Definition 1. This is investigated empirically in Figure 4. If the classifier does not exhibit such a preference, whitening likely has little effect.
Second, and crucially, the directions favored by this preference should align with spurious features. In this case, removing the preference can improve robustness.
Figure 4 suggests that the first property holds for MultiNLI, but this does not translate into a clear improvement in worst-group accuracy. One possible explanation we provide in the paper is that the high-variance directions in MultiNLI are less strongly aligned with spurious features than in the other benchmarks. We emphasize, however, that we have not directly measured this alignment.