Intrinsic Self-Distillation for Test-Time Adaptation

Test-time adaptation has emerged as a promising approach for addressing distribution shifts in target domains by improving predictions during inference on online data streams. For decades, the utilization of unlabeled data has been widely explored in semi-supervised and self-supervised learning, and provides a blueprint for test-time adaptation. Building on pioneering insights, recent solutions achieve notable improvements with data augmentation or auxiliary modules. However, such approaches demand additional computational complexity and reduce practical applicability. In this paper, we introduce intrinsic self-distillation for adapting to the target domain using intermediate representations, without augmentation or auxiliary modules. Our method leverages intermediate representations as intrinsic learning signals, motivated by two empirical observations: first, the outputs of deeper layers exhibit progressively higher accuracy. Second, predictions from intermediate representations are more robust to noise. We demonstrate the efficiency and effectiveness of our approach across various architectures and benchmarks.