acceptodds
Withdrawn from ICLR 2027

Transferability Estimation via Persistent Homology

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Selecting a pre-trained model for a downstream task is central to transfer learning, yet estimating transferability without fine-tuning remains challenging. We argue that this challenge reflects a deeper architectural question: whether the topology of a representation remains informative under domain shift. We study this question using 0-dimensional persistent homology. From this representation, we derive the Persistence Ratio (PR), a topology-based score that captures the balance between inter-class separation and intra-class variability in neural embeddings without the need for labels. Across ImageNet-pretrained CNNs and Vision Transformers evaluated on diverse downstream datasets, PR strongly correlates with fine-tuning performance, but only when topology is measured in the architecture-appropriate domain. Target-domain topology provides the most direct signal when it remains coherent. However, for Convolutional Neural Networks (CNNs), target embeddings often fragment before adaptation, making source-domain topology a more stable proxy for representation quality. For Vision Transformers (ViTs), by contrast, target embeddings preserve coherent cluster structure under domain shift, allowing topology measured directly on target data prior to fine-tuning to provide the most reliable estimates. This contrast shows that target-space transferability estimation is not universally reliable; its effectiveness depends on whether the architecture preserves meaningful target-domain topology before fine-tuning. These results reveal an architectural divide in representation geometry and position persistent homology as both a practical estimator for model selection and a diagnostic tool for understanding transferable representations.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Topological Data Analysis, Persistent Homology, Transfer Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rbFDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Persistence Ratio (PR), a transferability estimator based on zero-dimensional persistent homology of pretrained embeddings. PR characterizes cluster connectivity to rank candidate models before fine-tuning. Experiments on CNNs and ViTs report higher average correlations with downstream performance than the evaluated baselines, with source-domain PR performing better for CNNs and target-domain PR performing better for ViTs.

Critical strengths
  • The scoring rule is simple and requires neither fine-tuning nor sample-level labels, making it potentially useful when annotations or computational resources are limited.
  • The comparison between source- and target-domain topology offers an interesting perspective on representation structure under domain shift.
Critical weaknesses
  • Missing comparisons. The evaluation omits TTP [1], despite identifying it as the closest prior work, and relevant label-free baselines such as RankMe [2] and TranSAC [3]. Comparisons with competitive estimators including DISCO [4], LEAD [5], and ITM [6] would also help establish PR’s relative merits. These comparisons should distinguish supervision requirements, access to source data, and computational costs. Without them, the reported improvements provide limited evidence of competitiveness.

  • Limited evaluation. Evaluating CNNs and ViTs separately does not establish whether PR can reliably rank them within a shared candidate pool. A more informative evaluation would jointly include different architectures and pretraining paradigms, such as supervised learning, contrastive learning, and masked image modeling. ITM [6] provides a relevant reference protocol. This setting is particularly important because the preferred scoring domain differs between CNNs and ViTs in the reported experiments.

  • Unclear link to adaptability. PR measures unlabeled connectivity, but well-separated clusters need not align with the semantic distinctions required by the target task. Moreover, architectures and pretraining paradigms can exhibit different adaptation dynamics [6], motivating methods that model fine-tuning evolution [5,6]. Whether PR’s static score reliably predicts performance across these settings remains unclear. Although the memorization experiment illustrates one failure case, stronger controlled analyses on naturally pretrained models are needed to establish the method’s reliability and limitations.

[1] TTP: On Characterizing the Evolution of Embedding Space of Neural Networks Using Algebraic Topology. PRL, 2024.

[2] RankMe: RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank. ICML, 2023.

[3] TranSAC: TranSAC: An Unsupervised Transferability Metric Based on Task Speciality and Domain Commonality. Pattern Recognition, 2026.

[4] DISCO: Assessing Pre-Trained Models for Transfer Learning Through Distribution of Spectral Components. AAAI, 2025.

[5] LEAD: LEAD: Exploring Logit Space Evolution for Model Selection. CVPR, 2024.

[6] ITM: Implicit Modeling for Transferability Estimation of Vision Foundation Models. NeurIPS, 2025.

Questions for authors
  • What assumptions connect the unlabeled cluster structure measured by PR to the semantic distinctions required by the downstream task? What target-task information enters source-domain PR, and how does it distinguish suitability for different target tasks?

  • How does PR perform on a heterogeneous benchmark, such as the model pool evaluated in ITM [6], under matched fine-tuning protocols? Please report per-dataset ranking correlations, including weighted Kendall’s τ, alongside overall averages and scoring costs.

  • Does PR remain predictive across models with different adaptation dynamics, particularly contrastively pretrained and masked-image-modeling models? How should the scoring domain be selected for a mixed model pool without consulting downstream fine-tuning results?

Reviewer_hqs8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates transferability estimation for pretrained vision models without downstream fine-tuning. The authors propose the Persistence Ratio (PR), a metric based on 0-dimensional persistent homology that characterizes the global connectivity of feature representations through inter-cluster separation and intra-cluster compactness.

Experiments across CNNs and Vision Transformers demonstrate that PR achieves competitive correlations with downstream fine-tuning performance. The authors further identify architecture-dependent differences under domain shift: source-domain topology appears more informative for CNNs, while target-domain topology is more effective for ViTs. Additional experiments investigate the relationship between topology, semantic learning, and sampling stability.

Critical strengths

S1. Practical relevance. Transferability estimation is a meaningful problem that can reduce the computational cost of selecting and fine-tuning pretrained models.

S2. Interesting topological perspective. The use of persistent homology provides an interpretable perspective on representation geometry. The observed differences between CNNs and ViTs under domain shift are potentially interesting for understanding transferable representations.

S3. Broad experimental evaluation. The paper evaluates multiple CNN and ViT architectures across diverse downstream datasets, with additional analyses of topology, memorization, and sampling stability.

Critical weaknesses

W1. Limited analysis of downstream adaptation settings. Downstream transfer performance can be significantly influenced by adaptation strategies and hyperparameter configurations. Since the evaluation primarily relies on existing fine-tuning benchmarks, the robustness of PR across different adaptation protocols (e.g., linear probing vs. full fine-tuning) and hyperparameter settings remains unclear.

W2. Limited diversity of downstream tasks. Although the experiments cover multiple datasets, they primarily focus on visual classification tasks. The effectiveness of PR on classification tasks in other modalities, such as text or audio, remains unexplored. Furthermore, its applicability to other task types, such as object detection and semantic segmentation, has not been established.

W3. Limited diversity of pretrained models and training paradigms. The evaluation mainly focuses on conventional CNNs and Transformers with ImageNet-based pretraining, with only limited coverage of self-supervised models. Experiments involving more diverse pretraining paradigms and modern large-scale vision foundation models, such as CLIP, DINOv2, and MAE, would provide stronger evidence of the method's generalizability.

Questions for authors

Q1. Sensitivity to downstream adaptation. Different adaptation strategies (e.g., linear probing vs. full fine-tuning) and hyperparameter settings may lead to different model rankings. Have the authors investigated the stability of PR under these variations? To what extent does its predictive ability depend on the specific fine-tuning protocol?

Q2. Generalization across modalities and task types. Can PR effectively generalize to classification tasks beyond vision, such as text or audio classification? Furthermore, is the current formulation based on inter-cluster separation and intra-cluster compactness applicable to tasks such as object detection and semantic segmentation, or would task-specific modifications be necessary?

Q3. Generalization across pretraining paradigms and model scales. Models pretrained with different objectives, such as CLIP, DINOv2, and MAE, may exhibit different representation geometries. Do the authors expect PR to remain effective for these models and larger-scale vision foundation models? What potential limitations might arise?

Reviewer_DE7UOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies transferability estimation: ranking pre-trained models by expected fine-tuning performance without fine-tuning. It proposes the Persistence Ratio (PR), a label-free score built from 0-dimensional persistent homology (Vietoris–Rips filtration) of penultimate-layer embeddings. PR compares the persistence of inter-cluster merges with that of intra-cluster merges, so tight, well-separated clusters give a better score. The paper further argues that the domain in which PR should be computed depends on the architecture: source-domain (ImageNet) PR works best for CNNs, and target-domain PR works best for ViTs.

Critical strengths

The question of whether representation geometry should be measured on the source or the target domain is meaningful. The method is simple (computed once, on a single layer) and easy to reproduce. The analysis is reasonably thorough.

Critical weaknesses
  1. Limited gains and a missing key comparison. On ViTs, PR_t (0.579) vs. LogME (0.543) and SFDA (0.527) have heavily overlapping CIs, and Appendix D.2 itself acknowledges uncertainty in the relative ordering of methods. The closest prior work, TTP (Suresh et al., 2024), is described as "effective" in the related work, yet it is not compared against and no justification is given. Its code is publicly available, so a comparison at least on the CNN benchmark appears feasible.
  2. The "architectural divide" is not cleanly validated. The CNN and ViT results use different model zoos, datasets and training regimes, so the difference cannot be attributed to architecture itself. The evidence is also asymmetric: the gap is large for CNNs (PR_s 0.658 vs. PR_t 0.316), but small for ViTs (PR_t 0.579 vs. PR_s 0.542, with overlapping CIs).
Questions for authors
  1. TTP's implementation is public. Could you compare against it at least on the CNN benchmark? If differences in protocol make it non-comparable (TTP was originally evaluated with Pearson correlation on a small set of CNNs), please state this in the paper.
  2. Section 3.5 states that the informative domain is architecture-dependent, but this is inferred from the results in Section 4. For a new architecture, without ground-truth rankings, is there a diagnostic available before fine-tuning (e.g., a measure of target-embedding fragmentation) that tells a practitioner whether to use PR_s or PR_t?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.