SHARP: Spherical-Harmonic-Inspired Distribution Modeling and Prototype Recalibration for Test-Time Adaptation

Online test-time adaptation (TTA) adapts pre-trained vision-language models (VLMs) to distribution shifts using only an unlabeled test stream. Cache-based methods retain past test samples as instance-level evidence, but a finite cache covers only a limited portion of the target distribution. Class-conditional distributions provide a complementary global view, yet their statistics are hard to estimate from few observations in the high-dimensional CLIP embedding space, especially covariance estimation. We propose SHARP, which estimates class-wise distributions in a compact angular feature space. Motivated by the hyperspherical geometry of normalized CLIP embeddings, SHARP builds a spherical-harmonic-inspired representation from learnable directional projections and low-order angular functions, reducing the number of covariance parameters and improving estimation reliability. The resulting class statistics drive both distribution-aware scoring and variance-adaptive prototype recalibration. SHARP further combines these statistics with instance-level evidence from multiple caches and applies lightweight residual adaptation to the classifier and cache features while keeping the CLIP encoders frozen. Across 10 cross-dataset benchmarks, SHARP improves average accuracy over the strongest baseline by 0.97% and 1.16% with ResNet-50 and ViT-B/16, respectively. On ImageNet and its four distribution-shift variants, the corresponding gains are 0.94% and 0.95%.