Beyond Fixed Geometry: Reliable Test-Time Adaptation of Vision-Language Models

Under test-time distribution shift, vision–language models can fail along two distinct reliability axes: the pretrained similarity geometry may misweight target- relevant feature directions, while augmented views may remain confidently wrong. We introduce a composite heteroscedastic reference model whose MAP rule exposes two distinct reliability roles: directional comparison precision and view-specific observation precision. Guided by these roles, GeoRel (Geometry– Reliability Factorized Test-Time Adaptation) learns a shared positive-semidefinite comparison geometry from weak cross-modal class evidence through discrimina- tive attraction, separation, and redundancy conditioning, while ranking augmented observations through scale-free agreement between frozen-space query consistency and adaptive predictive certainty. EMA separately stabilizes noisy target-visual evidence used by the geometry-adapted prediction. Across natural-shift and cross- dataset benchmarks with RN50 and ViT-B/16, GeoRel achieves strong and broadly competitive performance. The resulting principle is simple: adapt the comparison geometry, but anchor view reliability in frozen space.