EXACT: Estimation–Support Asymmetry in Test-Time Adaptation

Test-time adaptation (TTA) uses unlabeled target observations to improve model performance under distribution shift, often by estimating target-side statistics from the observed data. Existing methods mainly focus on how these statistics should be estimated or optimized, while paying less attention to whether the available target data are sufficient for reliable estimation. Different target statistics, however, have different data requirements. Global statistics are estimated from all observed target samples, whereas class-specific statistics depend on samples from the corresponding classes. We refer to this difference as estimation–support asymmetry. Our analysis and experiments show that their relative reliability varies with the amount of available target data: global statistics are more reliable when data are limited, whereas class-specific statistics become more effective as target samples accumulate. These findings motivate an inference strategy that ties the use of target statistics to the data available for their estimation. We instantiate this principle in EXACT, a framework that uses global statistics when target data are limited and enables target prototype estimation as observations accumulate. A label-free, coverage-based criterion derived from our class coverage analysis determines when class-specific statistics are used. EXACT requires no source data, target labels, backpropagation, or updates to the pretrained model. Across seven corruption and natural-shift benchmarks, EXACT achieves the highest average accuracy among the compared methods in all four backbone/inference settings. Reliability and ablation analyses provide further evidence for the estimation–support perspective and its coverage-based implementation.