Who Wins Where? Conformal Model Comparison for Local Superiority

Standard model comparison is global, aggregating losses across the covariate space to declare a single winner. This can obscure heterogeneous performance, where different models are preferable in different regions. We introduce conformalized local model comparison, a split-sample framework for constructing calibrated local winner maps. Under exchangeability, our goal is to provide finite-sample, distribution-free control of the marginal false winner rate while allowing declarations to depend on covariates; the guarantee is marginal because pointwise conditional control is unattainable in this setting. Given a model comparison score, such as the difference between two squared losses, the method estimates local centers and scales from out-of-sample scores and conformally calibrates the remaining uncertainty. At a test point, the procedure declares a local winner only when a one-sided conformal upper bound falls below zero. We instantiate the framework with a pretrained TabPFN localizer whose regression head is fine-tuned to maximize conformal gain. For any choice of localizer, we prove that the error control applies to one-sided false declarations on the realized future comparison score. We further give sufficient conditions for pointwise recovery of the expected winner away from tie boundaries, show that aggregate comparison can sharply contradict the prevalence of local superiority, and derive a squared-loss bias–variance decomposition of local wins. Synthetic and real-world experiments show that the method recovers heterogeneous winner regions and abstains under uncertainty, identifying local gains that global comparison can miss. We provide our code and data in the supplementary material.