Transferability Estimation via Persistent Homology

Selecting a pre-trained model for a downstream task is central to transfer learning, yet estimating transferability without fine-tuning remains challenging. We argue that this challenge reflects a deeper architectural question: whether the topology of a representation remains informative under domain shift. We study this question using 0-dimensional persistent homology. From this representation, we derive the Persistence Ratio (PR), a topology-based score that captures the balance between inter-class separation and intra-class variability in neural embeddings without the need for labels. Across ImageNet-pretrained CNNs and Vision Transformers evaluated on diverse downstream datasets, PR strongly correlates with fine-tuning performance, but only when topology is measured in the architecture-appropriate domain. Target-domain topology provides the most direct signal when it remains coherent. However, for Convolutional Neural Networks (CNNs), target embeddings often fragment before adaptation, making source-domain topology a more stable proxy for representation quality. For Vision Transformers (ViTs), by contrast, target embeddings preserve coherent cluster structure under domain shift, allowing topology measured directly on target data prior to fine-tuning to provide the most reliable estimates. This contrast shows that target-space transferability estimation is not universally reliable; its effectiveness depends on whether the architecture preserves meaningful target-domain topology before fine-tuning. These results reveal an architectural divide in representation geometry and position persistent homology as both a practical estimator for model selection and a diagnostic tool for understanding transferable representations.