Let Classes Compete: Discriminative Cache Matching for CLIP Test-Time Adaptation

Cache-based test-time adaptation (TTA) improves CLIP under distribution shifts by retaining previous test images and reusing them as visual references. Orthogonal to how such caches are constructed, we study a complementary question: once an observation has been stored, how should a newly arriving test image retrieve useful evidence from it? We identify two limitations of conventional cache matching. First, cache membership is determined by CLIP's image–text predictions, whereas subsequent retrieval relies on full-space image–image similarity, so even correctly retained references can be misranked. Second, correcting the global comparison geometry still represents each image with a single pooled feature, which can overlook localized class evidence. We propose Discriminative Cache Matching (DCM), a training-free plug-in that preserves the pseudo-labeling and image-cache update rules of the underlying adapter while improving retrieval at both levels. Globally, DCM uses a fixed text-induced subspace to guide image–image cache matching. Locally, it reuses ViT patch tokens from the same retained images to perform fine-grained, class-competitive patch–patch retrieval. We further provide a conditional margin analysis characterizing when projection and re-normalization can correct misleading cache-reference rankings. Experiments on cross-dataset and out-of-distribution benchmarks demonstrate the effectiveness of DCM.