RELIABILITY-GATED TEST-TIME ADAPTATION FOR OPEN-WORLD VISION-LANGUAGE RECOGNITION

Pretrained vision-language models provide strong zero-shot recognition but remain brittle when deployment streams combine evolving domain shifts with categories absent from the task label set. Existing test-time adaptation methods typically update on confident predictions in a closed label space, making open-world streams especially vulnerable to confirmation bias and irreversible drift. We present REG TTA, a reliability-gated framework for online adaptation of frozen vision-language models. REG-TTA scores each incoming sample using prediction consistency across stochastic augmentations, image-text semantic agreement, and local feature density in a dual-timescale prototype memory. Only reliable known-class instances update a lightweight prompt adapter, while uncertain or potentially unknown inputs are excluded from entropy minimization and organized into temporary semantic prototypes. We analyze the gradient induced by unknown-class samples and show that closed-set entropy minimization introduces a systematic pseudo class bias whenever the model is overconfident on open-set inputs. The released supplementary package contains an implementation,public-dataset stream builders, aggregation scripts, and plotting utilities for ImageNet-C, ImageNet-R, ImageNet A, ImageNet-V2, ObjectNet, and DomainNet evaluation.