- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces TE-Bench, a unified benchmark that reevaluates more than 30 transferability scores, including 22 published methods and several simple baselines, across 44 pretrained vision checkpoints and 11 downstream tasks. Beyond consolidating existing evaluations, it systematically studies transferability estimation under different architectures, limited target data, source–target distribution shifts, heterogeneous model hubs, and fine-tuning protocols. The experiments show that no published estimator is consistently best, many sophisticated methods fail to beat simple signals such as parameter count or frozen-feature separability, and their reliability often drops under data scarcity and distribution shift. Based on these observations, the paper proposes TE-Ref, a simple combination of LDA-based feature separability and model capacity that outperforms the strongest published estimator on average, while also highlighting that transferability scores should be viewed as protocol-dependent screening tools rather than universal predictors of downstream performance.
Critical strengths
- The evaluation is broader and more consistent. Instead of testing one new score on a hand-picked model hub, TE-Bench places many estimators under a shared protocol covering CNNs, ViTs, and both supervised and self-supervised models.
- It offers practical insights rather than just another leaderboard result. The paper shows when TE methods fail under data scarcity, distribution shift, and protocol changes, and introduces TE-Ref as a simple but strong baseline that future methods should beat.
Critical weaknesses
- The ground truth is not fully standardized and may be noisy. It reuses fine-tuning results from different papers, while model rankings can change with the optimizer, schedule, and hyperparameter-search budget, so protocol differences are mixed into the evaluation,and the ground-truth rankings may be noisy because repeated fine-tuning runs are unavailable.
- The conclusions—and even TE-Ref's weighting strategy—are derived from the particular model hubs, architectures, and classification tasks included in TE-Bench, but the benchmark does not provide a principled way to determine which estimator should be trusted for a genuinely new setting. Consequently, strong retrospective correlations on these hubs may not translate to new domains, task types, model families, or fine-tuning protocols, leaving practitioners with limited actionable guidance when the deployment scenario falls outside the benchmark.
Questions for authors
- Intuitively, transfer performance should improve as the source and target tasks become more closely related and as the model becomes larger. This seems broadly consistent with TE-Bench's analysis. Does this mean that transferability estimation is, at its core, trying to quantify these two factors: source–target relatedness and model capacity?
8 comments
Question (relatedness and capacity). Thank you for this insightful question. The core of this paper is a comprehensive analysis that evaluates TE scores in a consistent setup, with the same model and target hubs for every score, under different criteria such as target distribution shift and data scarcity.
Within this analysis, our experiments point to two complementary signals behind the fine-tuned ranking, in line with the reviewer's intuition. The first is model capacity, which reflects the gain a model can obtain from fine-tuning. The second is the separability of the target features, which reflects how well the existing representation suits the target and which we use as a proxy for source-target relatedness. In the current experimental setup of the literature, the best published TE scores do not surpass TE-Ref, a simple combination of the two. On the self-supervised hub, where capacity is fixed, separability alone orders the models ().
However, as we point out in the fine-tuning protocols paragraph (line 423) and in the limitations (line 477), the fine-tuned ranking also depends on the protocol, so these two factors do not determine it fully. Understanding transferability estimation more fully requires further ground truths and new fine-tuning experiments, which are GPU-intensive. We provide this comprehensive study as a basis for that investigation.
Response to Question (relatedness and capacity)
Thank you for the clarification, though the response mainly reiterates observations within the existing benchmark setup rather than addressing the core concern directly: once a practitioner moves to a new domain, task type, model family, or fine-tuning protocol, how can they tell whether the paper's conclusions still hold and whether TE-Ref can still be trusted?
We thank the reviewer for the assessment and for recognising the practical value of the analysis.
W1 (ground truth). We thank the reviewer for raising this important point. None of our model hubs mixes ground truths. All models of a hub come from a single published ground truth, obtained under one protocol, and every τw is computed within one hub against that ground truth. The configurations, initialisations and pre-trained weights of each hub match the implementation details of its ground truth (Table 6).
On three of our four hubs, the ground truth already reduces run-to-run variation. The SFDA ground truth (Shao et al., 2022), used for the supervised CNN, self-supervised CNN and ViT hubs, repeats each fine-tuning three times and reports the average of these runs. The ITM ground truth (Zheng et al., 2025), used for the mixed hub, does not report run-to-run averaging.
We agree that the fine-tuning hyper-parameters affect transferability estimation, and Section 6 makes this point (Fine-tuning protocols, line 423). Our two ground truths were obtained under different fine-tuning protocols, and they share seven checkpoints with identical weights. One ground truth can therefore be used as a transferability score for the other. Their agreement per target is:
The ordering changes even among these seven checkpoints. The accuracies under each ground truth on two targets are below, with the rank among the seven in parentheses:
A frozen-feature score returns the same value under either protocol, so these hyper-parameters are not accounted for in the current TE literature. As noted in our limitations (line 477), this comparison covers seven checkpoints and one pair of protocols. Studying the effect properly requires new ground truths under a controlled experimental design.
W2 (choosing an estimator for a new setting). We thank the reviewer for this insightful point. TE-Ref needs no fine-tuning result: it combines two signals computed from the frozen checkpoint without any tuned parameter, so it applies to a new setup as it is. We also investigated where TE-Ref and the published TE scores fail (Tables 3 and 4). Section 6 (Transferability estimation in practice, line 451) turns this into guidance for a new setting: the shift of the target, the availability of labels, the size of the target set and the composition of the hub.
We did not create or modify the model and target hubs; they are the hubs studied in the literature. They already cover different regimes. On the self-supervised hub the capacity is fixed, so only the generalisation term affects the ranking. The mixed hub holds 20 different backbones, CNNs and ViTs trained with supervised and self-supervised objectives (Table 6). As a test outside the vision hubs, TE-Ref is also the strongest entry on the language hub (Appendix B.6, τw=0.789), which differs in modality, model family and task type.