Ordinal Compatibility for Prediction and Adaptation in Vision-Language Models

Online test-time adaptation gives each unlabeled sample two roles: it must be classified immediately, and it can contribute to the model state used for subsequent predictions. These roles call for different uses of the same evidence. A sample may require prediction correction while warranting a conservative update to the evolving class distributions. We introduce Prediction and Adaptation via Compatibility Evidence (PACE), a framework that uses ordinal compatibility between fixed semantic knowledge and adaptive distributional estimates to guide both decisions. For each sample, we compare the class rankings induced by text prototypes and online Gaussian models, and convert their rank displacements into a shared compatibility vector. At the class level, compatibility provides a log prior that adjusts dynamic predictions. At the sample level, its posterior-weighted expectation produces a continuous gate that controls the total responsibility assigned to Gaussian updates. This design connects prediction correction and selective adaptation through one relational signal while keeping their operations distinct. PACE requires no encoder updates or backpropagation. Under dataset-specific oracle tuning on full target streams, the full method improves top-1 accuracy over its rank-free counterpart in all 21 evaluations, with an average gain of 1.09 percentage points. The ordinal-only variant achieves a 1.02-point mean gain, identifying compatibility as the main source of improvement. The gate reduces the fraction of update mass assigned to incorrect classes in five of six diagnostic domains. Matched-mass controls favor compatibility selection over uniform attenuation in three of four domains, while a same-state diagnostic shows that corrected predictions can still carry substantial wrong-class allocation mass.