Two-Stage Isotropic Whitening for Source-Free Test-Time Adaptation

When deploying machine learning models in real-world scenarios, distribution shift poses a critical challenge. Test data often deviates from training distributions and severely degrades model performance. This issue becomes especially acute in source-free test-time adaptation (SF-TTA) because models must adapt to unlabeled target data without source data or annotations. To address this challenge, we introduce Two-Stage Isotropic Whitening (TSIW) to facilitate target feature learning. TSIW integrates Whitening Batch Normalization (WBN) with Whitening Contrastive Learning (WCL). At the feature level, WBN enforces isotropic feature distributions via ZCA whitening. This suppresses domain-specific covariance structures and improves stability under distribution shifts. At the representation level, WCL extends standard contrastive learning through global feature whitening. This eliminates redundant feature correlations and optimizes representation geometry on a hypersphere to preserve semantic relationships. Through this two-stage process, WBN handles low-level feature standardization while WCL refines global representation geometry. Together, they yield highly generalizable features. Extensive experiments demonstrate that TSIW achieves state-of-the-art performance on major benchmarks, including VisDA-C, DomainNet-126, ImageNet-C, and CIFAR-100-C.