Soft Pseudo Supervised Contrastive Learning for Test-Time Adaptation

Test-time adaptation (TTA) aims to refine pretrained models for shifted test distributions using unlabeled test inputs. Teacher–student self-training shows state-of-the-art performance by minimizing the cross-entropy loss with the pseudo-labels generated by the teacher. However, cross-entropy-based approaches remain vulnerable to noisy pseudo-labels despite the use of various regularization methods. In this work, we propose Pseudo Supervised Contrastive Learning (PSCL) as an alternative to cross-entropy that is a better suited loss for noisy labels in TTA. PSCL uses teacher's pseudo-labels to pull together same-class pair representations of test samples while pushing the rest apart. We propose two variants of PSCL. Hard PSCL assigns a binary relation according to whether the teacher predicts two pairs to be the same class, whereas Soft PSCL weights each relation by the teacher-estimated probability that the two samples belong to the same class. Specifically, for each anchor, PSCL aligns the student’s similarity distribution over other samples with the teacher-estimated probabilities that those samples belong to the same class as the anchor. Our theory provides a guarantee that, under mild conditions, soft pairwise pseudo-labels better approximate the true pairwise labels than hard pseudo-labels. We show that PSCL, as a sole objective, outperforms state-of-the-art baselines including methods that combine cross-entropy with weak contrastive regularization and other regularization techniques. This result holds across various experiments on the standard continual TTA (CTTA) benchmarks CIFAR-10-C, CIFAR-100-C, and ImageNet-C.