AdaEval: Adaptive Importance Weighting for Reliable Model Evaluation under Unknown Covariate Shift

Reliable model evaluation in the wild often relies on the AUTOEVAL framework, which augments a small human-labeled dataset with massive synthetic labels to produce low-variance performance estimates. A critical limitation is that AUTOEVAL assumes either that the labeled and unlabeled data share the same distribution or that the true importance weights are known, both of which are unrealistic under real-world covariate shift. We propose ADAEVAL, a method that learns importance weights directly from unlabeled data by training a discriminative classifier to distinguish labeled from target inputs. The resulting weighted estimator re-targets the labeled sample to the shifted distribution without any prior knowledge of the shift. We prove consistency and asymptotic normality under mild conditions on the weight estimation error. On ImageNet with simulated covariate shift, standard AUTOEVAL collapses to a coverage of just $0.017$ with $500$ labeled points; ADAEVAL achieves $0.81$–$0.87$ coverage against PPI++'s $0.77$–$0.84$, and reduces MSE by $45$–$56\%$ relative to classical estimation. On ProteinGym, under a strong fitness-biased shift, ADAEVAL reduces absolute MSE bias by $46$–$60\%$ while coverage decays to near zero as $n$ grows large, a residual gap we characterize explicitly. ADAEVAL reduces bias and MSE under unknown covariate shift, with a fully characterized residual coverage gap; it is not a general solution for statistically valid confidence intervals across all regimes.