Do Not Cross the Boundary: Within-Class Whitening for Test-Time Adaptation

Test-time adaptation (TTA) recalibrates a deployed model under domain shift using only unlabeled test data, and recent methods have substantially improved the robustness of such models. However, most existing TTA methods adjust only class-agnostic statistics and overlook the distortion of the feature distribution within each class. Concretely, domain shift elongates the covariance of each class, but existing TTA methods leave it elongated even after adaptation, causing some samples to cross the decision boundary and thus remain misclassified. To mitigate this, we propose Within-Class Whitening (WCW), a plug-in method that whitens the penultimate features with the class-averaged within-class covariance, without backpropagation or pre-computed source statistics. We analyze the family of affine feature transformations and show that WCW makes the within-class covariance isotropic, which lowers an upper bound on the probability of crossing the decision boundary under domain shift. We also identify that restricting the whitening scope to the within-class component avoids the implicit injection of a uniform class prior caused by the rearrangement of the class means. Extensive experiments on various datasets, backbones, and adaptation scenarios show that WCW consistently improves representative parameter-updating TTA methods, outperforms existing TTA plug-ins in most cases, and substantially stabilizes the adaptation procedure.