ShadowTransfer: A Geographic Transfer Benchmark for Overhead Shadow Detection

Overhead aerial imagery feeds dense-prediction pipelines that are routinely trained in one geographic region and deployed in another, yet no benchmark measures how shadow detection models transfer across regions. Existing overhead shadow datasets either lack a geographic transfer protocol or rely on automatically generated labels. We introduce \shadow, a benchmark of $4,500$ human-verified shadow masks across three U.S. cities at two native resolutions, paired with a leave-one-city-out evaluation protocol. Across three architectures and seven adaptation methods, all adaptation methods degrade under geographic transfer. We propose a three-probe framework that diagnoses where and how the transfer breaks: the failure operates in feature space (not pixel space), shows up as a confidence collapse on shadow pixels while background confidence stays intact, and is recoverable by retraining different architectural components depending on the model: encoder for some, decoder for others. These findings replicate on INRIA building footprint extraction under the same protocol, suggesting they describe geographic transfer broadly rather than something specific to shadows. Guided by the diagnosis, we build Spectral Information Bottleneck (SIB), a plug-in module with one component per finding. Across the $9$ (architecture, held-out-city) cells, attains the highest mean mIoU ($70.92$) and the smallest worst-case gap to the within-city upper bound ($7.18$ mIoU). The benchmark, LOCO protocol, code, and trained models are released to support reproducible evaluation of geographic transfer in overhead dense prediction.