acceptodds
Under review as a conference paper at ICLR 2027

TE-Bench: A Unified Benchmark for Transferability Estimation

Abstract

Transferability estimation (TE) predicts which pre-trained checkpoint will achieve the best post-fine-tuning performance on a downstream target task without fine-tuning any of the candidate checkpoints, turning a model search that would cost GPU hours to days of computation into one that runs in seconds to minutes. We built TE-BENCH, a unified evaluation of transferability estimation that covers published TE scores, several experimental setups, and a set of baselines that are rarely compared in the literature. We found that some of these baselines outperform the published TE scores, and we introduce a simple reference baseline (TE-Ref) that significantly outperforms them in the overall setting, which covers supervised and self-supervised vision models across both CNN and ViT architectures. TE-BENCH covers more than transferability estimation scores, distinct checkpoints, and target tasks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
model selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1ASbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces TE-Bench, a unified benchmark that reevaluates more than 30 transferability scores, including 22 published methods and several simple baselines, across 44 pretrained vision checkpoints and 11 downstream tasks. Beyond consolidating existing evaluations, it systematically studies transferability estimation under different architectures, limited target data, source–target distribution shifts, heterogeneous model hubs, and fine-tuning protocols. The experiments show that no published estimator is consistently best, many sophisticated methods fail to beat simple signals such as parameter count or frozen-feature separability, and their reliability often drops under data scarcity and distribution shift. Based on these observations, the paper proposes TE-Ref, a simple combination of LDA-based feature separability and model capacity that outperforms the strongest published estimator on average, while also highlighting that transferability scores should be viewed as protocol-dependent screening tools rather than universal predictors of downstream performance.

Critical strengths
  • The evaluation is broader and more consistent. Instead of testing one new score on a hand-picked model hub, TE-Bench places many estimators under a shared protocol covering CNNs, ViTs, and both supervised and self-supervised models.
  • It offers practical insights rather than just another leaderboard result. The paper shows when TE methods fail under data scarcity, distribution shift, and protocol changes, and introduces TE-Ref as a simple but strong baseline that future methods should beat.
Critical weaknesses
  • The ground truth is not fully standardized and may be noisy. It reuses fine-tuning results from different papers, while model rankings can change with the optimizer, schedule, and hyperparameter-search budget, so protocol differences are mixed into the evaluation,and the ground-truth rankings may be noisy because repeated fine-tuning runs are unavailable.
  • The conclusions—and even TE-Ref's weighting strategy—are derived from the particular model hubs, architectures, and classification tasks included in TE-Bench, but the benchmark does not provide a principled way to determine which estimator should be trusted for a genuinely new setting. Consequently, strong retrospective correlations on these hubs may not translate to new domains, task types, model families, or fine-tuning protocols, leaving practitioners with limited actionable guidance when the deployment scenario falls outside the benchmark.
Questions for authors
  • Intuitively, transfer performance should improve as the source and target tasks become more closely related and as the model becomes larger. This seems broadly consistent with TE-Bench's analysis. Does this mean that transferability estimation is, at its core, trying to quantify these two factors: source–target relatedness and model capacity?
8 comments

We thank the reviewer for the assessment and for recognising the practical value of the analysis.

W1 (ground truth). We thank the reviewer for raising this important point. None of our model hubs mixes ground truths. All models of a hub come from a single published ground truth, obtained under one protocol, and every τw\tau_w is computed within one hub against that ground truth. The configurations, initialisations and pre-trained weights of each hub match the implementation details of its ground truth (Table 6).

On three of our four hubs, the ground truth already reduces run-to-run variation. The SFDA ground truth (Shao et al., 2022), used for the supervised CNN, self-supervised CNN and ViT hubs, repeats each fine-tuning three times and reports the average of these runs. The ITM ground truth (Zheng et al., 2025), used for the mixed hub, does not report run-to-run averaging.

We agree that the fine-tuning hyper-parameters affect transferability estimation, and Section 6 makes this point (Fine-tuning protocols, line 423). Our two ground truths were obtained under different fine-tuning protocols, and they share seven checkpoints with identical weights. One ground truth can therefore be used as a transferability score for the other. Their agreement per target is:

Airc.Cal101CarsC10C100DTDFlwrFoodPetsSUNVOCAvg
τw\tau_w (SFDA vs ITM)0.8940.7980.8350.7010.9600.8140.8520.8370.8250.8690.5510.812

The ordering changes even among these seven checkpoints. The accuracies under each ground truth on two targets are below, with the rank among the seven in parentheses:

ModelFlowers-102 (SFDA)Flowers-102 (ITM)VOC2007 (SFDA)VOC2007 (ITM)
ResNet-3495.20 (7)90.03 (4)84.60 (4)81.56 (3)
DenseNet-12197.02 (3)90.41 (3)85.28 (3)82.75 (1)
DenseNet-16997.32 (1)91.58 (1)85.77 (1)81.85 (2)
DenseNet-20197.10 (2)91.07 (2)85.67 (2)80.86 (4)
MnasNet-1.095.39 (6)76.48 (7)81.06 (7)78.78 (7)
GoogLeNet95.76 (4)88.01 (5)82.58 (6)80.45 (5)
Inception-v395.73 (5)87.14 (6)83.84 (5)79.50 (6)

A frozen-feature score returns the same value under either protocol, so these hyper-parameters are not accounted for in the current TE literature. As noted in our limitations (line 477), this comparison covers seven checkpoints and one pair of protocols. Studying the effect properly requires new ground truths under a controlled experimental design.

W2 (choosing an estimator for a new setting). We thank the reviewer for this insightful point. TE-Ref needs no fine-tuning result: it combines two signals computed from the frozen checkpoint without any tuned parameter, so it applies to a new setup as it is. We also investigated where TE-Ref and the published TE scores fail (Tables 3 and 4). Section 6 (Transferability estimation in practice, line 451) turns this into guidance for a new setting: the shift of the target, the availability of labels, the size of the target set and the composition of the hub.

We did not create or modify the model and target hubs; they are the hubs studied in the literature. They already cover different regimes. On the self-supervised hub the capacity is fixed, so only the generalisation term affects the ranking. The mixed hub holds 20 different backbones, CNNs and ViTs trained with supervised and self-supervised objectives (Table 6). As a test outside the vision hubs, TE-Ref is also the strongest entry on the language hub (Appendix B.6, τw=0.789\tau_w = 0.789), which differs in modality, model family and task type.

Question (relatedness and capacity). Thank you for this insightful question. The core of this paper is a comprehensive analysis that evaluates TE scores in a consistent setup, with the same model and target hubs for every score, under different criteria such as target distribution shift and data scarcity.

Within this analysis, our experiments point to two complementary signals behind the fine-tuned ranking, in line with the reviewer's intuition. The first is model capacity, which reflects the gain a model can obtain from fine-tuning. The second is the separability of the target features, which reflects how well the existing representation suits the target and which we use as a proxy for source-target relatedness. In the current experimental setup of the literature, the best published TE scores do not surpass TE-Ref, a simple combination of the two. On the self-supervised hub, where capacity is fixed, separability alone orders the models (τw=0.734\tau_w = 0.734).

However, as we point out in the fine-tuning protocols paragraph (line 423) and in the limitations (line 477), the fine-tuned ranking also depends on the protocol, so these two factors do not determine it fully. Understanding transferability estimation more fully requires further ground truths and new fine-tuning experiments, which are GPU-intensive. We provide this comprehensive study as a basis for that investigation.

Reviewer_1ASbOpenReview ↗

Response to Question (relatedness and capacity)

Thank you for the clarification, though the response mainly reiterates observations within the existing benchmark setup rather than addressing the core concern directly: once a practitioner moves to a new domain, task type, model family, or fine-tuning protocol, how can they tell whether the paper's conclusions still hold and whether TE-Ref can still be trusted?

Reviewer_9UPPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the inconsistent evaluation settings across existing transferability estimation studies, this paper introduces TE-Bench, a unified benchmark for comparing different approaches. The benchmark covers both supervised and self-supervised pretrained models across 11 target datasets. Additionally, the paper proposes TE-Ref, a simple baseline that combines parameter counts and linear separability scores, achieving performance comparable to or even better than existing state-of-the-art methods. Finally, the paper investigates the effects of domain shifts and dataset sizes, as well as the computational costs of the compared methods.

Critical strengths
  • The analysis of domain shift is interesting, and the experimental design of separating high-shift and low-shift scenarios is well-motivated.
Critical weaknesses
  • Although the proposed TE-Ref achieves promising results on TE-Bench, it considers only a limited set of factors relevant to transferability estimation. In particular, its reliance on parameter counts may lead to unreliable predictions in certain scenarios. For example, a very large randomly initialized model could receive a high score simply due to its parameter count, despite having poor transferability. While I appreciate the simplicity and effectiveness of TE-Ref, its design lacks a more comprehensive consideration of transferability factors, raising concerns about whether its strong performance generalizes beyond TE-Bench or is specific to the benchmark settings.

  • The evaluation primarily focuses on methods that rely on features extracted from frozen pretrained backbones, without explicitly accounting for differences in fine-tuning protocols. However, fine-tuning strategies, including optimizer choices, learning rate schedules, and hyperparameter configurations, can substantially affect downstream performance and model rankings. Consequently, the adopted evaluation setting may not fully capture these factors, potentially limiting the generalizability and practical applicability of the proposed benchmark and its findings.

Questions for authors

Besides the weakness shown in the above section, please also see the following questions:

Q1: In Table 3, TE-Ref appears to exhibit the largest performance drop among the compared methods when moving from low-shift to high-shift domains. What could explain this observation, given that TE-Ref incorporates a separability score that should, to some extent, capture the effects of domain shift?

Q2: Weight-only scores do not account for the characteristics of downstream datasets. Would it be beneficial to combine these scores with other transferability estimation methods to capture a broader range of factors that influence transferability?

2 comments

We thank the reviewer for the comments.

W1 (reliance on parameter count). We would like to highlight that we use the parameter count as a simple proxy for model capacity, because it suits the current setup. Capacity in the broader sense, the performance a model can reach, depends jointly on its size, its pre-training data and the compute spent on its training (Kaplan et al., 2020; Zhai et al., 2022). For vision transformers, the amount of pre-training data, the augmentation and the regularisation also change the result substantially (Steiner et al., 2022).

In our hubs, every checkpoint is a competitive model, fully trained with its published recipe. All are pre-trained on ImageNet: ImageNet-1k, or ImageNet-21k followed by ImageNet-1k for three ViTs (Table 6). Among such models, the parameter count is the simplest signal of how their capacity differs. A randomly initialised network has the parameter count of its architecture but none of the optimisation or pre-training that capacity includes, so the proxy does not apply to it.

Weight-based metrics can also serve as a capacity proxy, since they are computed from the trained weights rather than from the architecture alone. The two we study behave differently:

  • The effective rank of Eq. (8) follows the parameter count closely, with a Kendall τ\tau of 0.53 on supervised CNNs and 0.67 on ViTs. When combined with TE scores, its gains match those of the parameter count (see our answer to Q2).
  • α\alpha-weighted is only weakly related to size, with a Kendall τ\tau of 0.38 and 0.00, and reflects how well a model was trained. It is the strongest single score on supervised CNNs (τw=0.721\tau_w = 0.721), but it reverses on self-supervised backbones (Section 6), where its premise does not hold.

TE-Ref weights capacity and generalisation by each term's share of the two standard deviations over the hub, so a signal that varies more across the models counts for more. When capacity is fixed, the capacity term has zero standard deviation and its weight vanishes, so TE-Ref ranks the models by generalisation alone. This is the case on our self-supervised hub, where the backbone is the same ResNet-50 and only the pre-trained weights differ.

This is the standard and practical setting of transferability estimation covered in the literature, and we did not remove or skip any reported benchmark setting. In a real scenario, a practitioner selects among valid, compatible pre-trained checkpoints rather than untrained ones, and screening out invalid models is not the goal of this work. The goal is to show that the best published TE scores do not surpass this simple reference baseline in their own benchmark settings. Beyond the vision hubs, TE-Ref is also the strongest entry on the language hub (Appendix B.6, τw=0.789\tau_w = 0.789), where the models, the tasks, and the modality all differ.

W2 (fine-tuning protocols). We agree that the fine-tuning hyper-parameters affect transferability estimation, and Section 6 makes this point (Fine-tuning protocols, line 423). We use two ground truths (Shao et al., 2022; Zheng et al., 2025) obtained under different fine-tuning protocols. Seven checkpoints appear in both with identical weights: ResNet-34, DenseNet-121, DenseNet-169, DenseNet-201, MnasNet-1.0, GoogLeNet and Inception-v3. One ground truth can therefore be used as a transferability score for the other. Their agreement per target is:

Airc.Cal101CarsC10C100DTDFlwrFoodPetsSUNVOCAvg
τw\tau_w (SFDA vs ITM)0.8940.7980.8350.7010.9600.8140.8520.8370.8250.8690.5510.812

The ordering changes even among these seven checkpoints: their agreement varies from 0.960 on CIFAR-100 to 0.551 on VOC2007 (Section 6, Fine-tuning protocols, line 423). A score that reproduced one protocol exactly would therefore reach only τw=0.812\tau_w = 0.812 against the other. A frozen-feature score returns the same value under either protocol, so the current TE literature does not account for these hyper-parameters. TE-Ref itself holds under both protocols (Table 1). Under the SFDA protocol it is above the best published score on two hubs and 0.009 below it on the third, and under the ITM protocol it is 0.083 above.

As noted in our limitations (Section 6, lines 448 to 450, and line 477), this experiment covers seven checkpoints and a single pair of protocols. It also cannot separate the effect of the recipe from run-to-run variation, which would need repeated fine-tuning runs under each protocol. It therefore shows the direction of the effect rather than its full size. Studying it properly requires more shared checkpoints and new ground truths under a controlled experimental design that varies the optimiser, schedule and hyper-parameter search one at a time.

Q1 (drop under shift). We would first like to note that on the supervised CNN hub, TE-Ref's drop under shift is small (-0.034), and smaller than those of ITM (-0.082) and NCTI (-0.045).

On the ViT hub, the drop (-0.169) comes from how the generalisation and capacity terms interact. On low-shift targets they are complementary: TE-Ref (0.759) is above both LDA (0.422) and the parameter count (0.710), and above every other score listed for this hub in Table 3. On high-shift targets they are not, and TE-Ref (0.590) falls below the parameter count alone (0.639). The largest drop therefore starts from the highest low-shift value on this hub, and on high-shift targets the separability term adds less to the capacity term.

Specifically, the generalisation term is a linear separability measure. Following the standard protocol of the TE literature, it reads only the penultimate features (and the classifier output, when present) of the frozen model. On low-shift targets these features are closer to what fine-tuning produces. On high-shift targets fine-tuning has to change the earlier, feature-forming layers, and a score on the frozen penultimate features does not observe that change. This agrees with two prior results:

  • Yosinski et al. (2014) show that features move from general in the first layers to specific in the last ones, and that their transferability decreases as the distance between the base and target tasks grows.
  • Lee et al. (2023) show that the layers worth fine-tuning depend on the type of shift: for image corruptions, tuning only the first few layers works best.

The separability term therefore captures the shift only through the frozen features, not through the change that fine-tuning makes to them, and it is the part of TE-Ref that degrades under shift. LDA alone loses 0.309 on supervised CNNs, 0.160 on the mixed hub and 0.131 on self-supervised CNNs (Appendix A.6). On the self-supervised hub, TE-Ref ranks the models as LDA does, because the parameter count is constant there.

Q2 (combining weight-only scores with other TE scores). We thank the reviewer for the question of whether weight-only scores, which do not read the target data, add information when combined with target-based TE scores. On the three SFDA hubs, we combined each weight-only score WW (α\alpha-weighted and the effective rank of Eq. (8)) with five TE scores MM as λF(M)+(1−λ)F(W)\lambda F(M) + (1-\lambda) F(W). Here FF is the min-max normalisation over the hub and λ\lambda is chosen on the ground truth. This choice is supervised, so the results are an upper bound and are not comparable with TE-Ref, which uses no ground truth.

TE scoreAlone+ α\alpha-weighted+ Eff. rank+ Eff. rank, capacity removed+ #Params
LDA0.5490.5490.6950.5490.710
SFDA0.6490.6570.6690.6490.690
NCTI0.6070.6070.6880.6090.690
ITM0.5910.5910.6850.5910.678
H-Score0.5770.5770.6310.5770.619

(τw\tau_w averaged over the eleven targets and the three hubs, with one λ\lambda for all hubs.)

α\alpha-weighted helps only on supervised CNNs, for example H-Score from 0.596 to 0.812 with a per-hub λ\lambda. On the other two hubs, the best λ\lambda keeps the TE score alone. The effective rank helps on average, but the gain is capacity: the parameter count gives comparable gains, and the gain vanishes once the parameter count is regressed out of the effective rank. We also chose λ\lambda on the other two hubs, without the ground truth of the hub being evaluated. With that choice, α\alpha-weighted improves none of the five TE scores, and the effective rank improves LDA and NCTI and leaves the other three within 0.005.

The weight-only scores add little because their information overlaps with that of the TE scores. They order 19 to 49% of model pairs differently from the TE scores. Yet 75 to 87% of the pairs they order correctly are already ordered correctly by the TE score. Weight-only scores therefore add little beyond capacity, which TE-Ref already takes from the parameter count. Without any ground truth, TE-Ref reaches 0.701 on these hubs (Table 2), on par with the supervised combinations above. We will add this analysis to the appendix.

References [1] Kaplan et al. Scaling Laws for Neural Language Models. arXiv:2001.08361, 2020.

[2] Zhai et al. Scaling Vision Transformers. CVPR 2022.

[3] Steiner et al. How to Train Your ViT? Data, Augmentation, and Regularization in Vision Transformers. TMLR 2022.

[4] Yosinski et al. How Transferable Are Features in Deep Neural Networks? NeurIPS 2014.

[5] Lee et al. Surgical Fine-Tuning Improves Adaptation to Distribution Shifts. ICLR 2023.

[6] Shao et al. Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Space. ECCV 2022.

[7] Zheng et al. Implicit Modeling for Transferability Estimation of Vision Foundation Models. NeurIPS 2025.

Reviewer_gR5mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper raises a unified transferability estimation (TE) benchmark, which evaluates 44 pretrained models, 11 downstream task and 4 model families, across over 30 evaluations and baselines.

Authors find parameters, feature dimensions and other metrics usually approach or even surpass existing methods, and they propose a reference baseline (TE-Ref) that combines linear discriminant analysis (LDA) with the number of parameters. On the mixed model set, TE-Ref achieved a weighted Kendall’s τ of 0.572, outperforming the best published method’s 0.490.

The paper also analyzes the impact of distribution shift, few-shot scenarios, feature dimension bias, and fine-tuning protocols on estimation performance.

Critical strengths
  1. The evaluation is relatively comprehensive. It provides a unified comparison of various methods for estimating transferability. These cover different model architectures and pretrained methods, which help mitigate inconsistent experiment setup issue in existing research.

  2. The baseline analysis is valuable. It demonstrates that simple metrics, such as the number of parameters and feature separability, are highly competitive. These provide a low-cost reference for practical model selection.

  3. The analysis is thorough. In addition to overall rankings, the study examines the effects of data volume, distribution shifts, and model architectures. In particular, the paper insightfully find that feature dimension may interfere with H-Score and SFDA evaluation results

  4. The reproducibility materials are comprehensive. The provision of code, implementation details, and computational overhead analysis facilitates future research..

Critical weaknesses
  1. Singh et al. (2026), in their paper “How NOT to Benchmark Your SITE Metric,” have already pointed out that existing benchmarks suffer from unreasonable model spaces, and that simple heuristics may outperform complex estimators. This paper needs to more clearly articulate its new contributions relative to existing research. [1] Singh, P., Hess, S., & Vanschoren, J. (2026). How NOT to benchmark your SITE metric: Beyond static leaderboards and towards realistic evaluation. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2510.06448

  2. There may be issues with the data partitioning. In Table 7 on page 17, the sample sizes for CIFAR-10 and Food-101 are 60,000 and 101,000, respectively, which correspond to the full dataset sizes. Since some TE methods require target labels, it is essential to verify whether test set labels were used. Otherwise, data leakage may occur, affecting the validity of the evaluation. At this time, it cannot be confirmed whether leakage has occurred.

  3. The performance advantages lack sufficient statistical support. In all four Model Hubs, TE-Ref’s improvements over the respective strongest published methods did not reach statistical significance (p > 0.05) . Although the comparison with NCTI across Model Hubs reached statistical significance (p = 0.003), this is not sufficient to prove that TE-Ref is universally superior to the strongest methods in each scenario.

  4. The validation of the core method is insufficient. Table 33 on page 34 does not report the top recall rates (Recall@1/3) for TE-Ref. Furthermore, Appendix B.8 indicates that LDA exhibits in-sample evaluation bias, but the impact of cross-fitting on TE-Ref has not been verified.

  5. Limited generalizability. The experiments primarily used earlier visual models and lacked recent representative foundation models. Additionally, the ground truth performance data from different model hubs was derived from different fine-tuned protocols, which may have affected the comparison results.

Questions for authors
  1. Novelty: Compared to existing benchmark studies [1], what new findings does TE-Bench offer in addition to expanding the scope of evaluation? Can you more clearly clarify the unique contributions of this paper? [1] Singh, P., Hess, S., & Vanschoren, J. (2026). How NOT to benchmark your SITE metric: Beyond static leaderboards and towards realistic evaluation. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2510.06448

  2. Data Splits: The sample sizes for some datasets in Table 7 correspond to the full datasets. Were test labels used when calculating the TE scores? If so, can you provide experimental results based solely on the training set?

  3. Statistical Significance: TE-Ref does not significantly outperform the strongest published methods on any of the model hubs. How do the authors explain its overall advantage? Could you provide confidence intervals for comparisons with the strongest methods on each model hub, as well as more robust statistical tests?

  4. Method Validation: Could you provide TE-Ref’s optimal Recall@1/3 and Selection Regret? Given the finding in Appendix B.8 that LDA exhibits in-sample evaluation bias, would TE-Ref’s performance improve after applying cross-fitting?

  5. Generalizability: Can the main conclusions be validated on newer foundation models such as CLIP and DINOv2? Do different fine-tuning protocols affect the method rankings and the relative advantages of TE-Ref?

6 comments

We thank the reviewer for the careful reading and for recognising the breadth of the evaluation, the value of the baseline analysis and the reproducibility materials.

W1 and Q1 (relation to Singh et al.). Thank you for pointing us to Singh et al. (2026); we will cite and discuss it in the revision. TE-Bench addresses a different research question. Singh et al. ask whether conventional transferability benchmarks are a meaningful test of model selection. They show that static model rankings and capacity differences can inflate the apparent effectiveness of existing estimators, so their focus is how to construct a more discriminative benchmark. TE-Bench asks which estimators are reliable, under which conditions, and why.

To answer this, we go beyond identifying capacity as a confounding factor and examine what determines the fine-tuned ranking. Our experiments point to two complementary signals. The first is model capacity, which reflects the gain a model can obtain from fine-tuning. The second is the separability of the target features, which reflects how well the existing representation suits the target. This motivates TE-Ref, a simple, explicitly defined reference that combines the two without any fine-tuning.

The capacity term of TE-Ref contributes nothing when model size is fixed, so TE-Ref does not rely on model scale alone. Our self-supervised hub holds ten ResNet-50 checkpoints that differ only in their pre-trained weights. This is the kind of model space Singh et al. recommend (models with comparable computational budgets, their Best Practice 2), and on it a static ranking has nothing to offer. Separability orders these models: LDA, and therefore TE-Ref, reaches τw=0.734\tau_w = 0.734. The data-free α\alpha-weighted metric, the best score on the supervised CNN hub (0.7210.721), reverses there (−0.562-0.562).

Singh et al. evaluate six metrics on a single hub, the eleven supervised CNNs that form one of our four hubs, and on six of our eleven targets. TE-Bench also examines conditions they do not study:

  • low versus high source-target shift (Table 3);
  • target-data scarcity (Table 4);
  • the fine-tuning protocol behind the ground truth (Section 6).

Within these conditions, TE-Bench reports four findings. First, H-Score and SFDA partly measure the feature dimension rather than class separability. They rank the models almost as the width does (τw=0.763\tau_w = 0.763 and 0.6430.643 on supervised CNNs), and projecting all features to a common dimension lowers them from 0.596 to 0.469 and from 0.659 to 0.576, while LDA is unchanged (Appendix B.3). Second, most scores degrade on high-shift targets and with few samples per class (Tables 3 and 4), which a single average over targets hides. Third, the ground truth itself depends on the fine-tuning protocol: two published ground truths agree only at τw=0.812\tau_w = 0.812 on the same checkpoints (Section 6). Fourth, a simple reference, rather than a static ranking, can serve as the baseline that new TE scores must beat. A static ranking assigns the same order to every target, whereas TE-Ref adapts to the target and, when capacity is fixed, ranks the models by separability alone. Across hubs that differ in architecture, pre-training objective and composition, no published TE score significantly outperforms it (Table 1).

W2 and Q2 (data splits). At selection time, a practitioner wants to know which candidate checkpoint will reach the highest accuracy after fine-tuning, an accuracy typically reported on the test split. A transferability score answers this before any fine-tuning, from whatever labelled target data is available, so the train/test distinction does not matter at this stage. The split is needed only to measure the fine-tuned accuracy that serves as the ground truth.

No training takes place in the score calculation. Every score is computed from the frozen pre-trained checkpoint through a forward pass over the target data, and never sees a fine-tuned model, its predictions or its accuracy. The ground-truth accuracies come from the published tables of Shao et al. (2022) and Zheng et al. (2025) and were obtained independently of our scores. A score therefore has no access to the quantity it is evaluated against. The train, validation and test splits also come from the same target distribution, so including the test images enlarges the sample that a score reads without changing what it measures.

We computed every score on all available target data, comprising the train, validation and test splits. The official code of published scores differs on this point: SFDA (Shao et al., 2022), PED (Li et al., 2023), and NCTI (Wang et al., 2023) extract features on the concatenated train, validation and test splits. To compare all scores under a single protocol, we follow them. Every score and every model is evaluated on exactly the same images, so the split cannot favour one score over another, and the comparison between scores, which is what the benchmark reports, is unaffected. On the language hub (Appendix B.6), scored on training data only, TE-Ref is again the strongest entry (τw=0.789\tau_w = 0.789). In the revision, we will state the split in Appendix A.2 and add an appendix section with training-split-only scores for the vision hubs.

W3 and Q3 (statistical significance). We thank the reviewer for this comment. We agree that the pooled test does not show that TE-Ref is superior to the strongest method on every hub, and we do not make that claim. The manuscript states that the per-hub differences are small and that none is significant (lines 415 to 419).

As the reviewer notes in strength 2, TE-Ref is intended as a simple, low-cost reference. The purpose of the significance tests is to check whether any published TE score surpasses it. On each hub we pair TE-Ref with the strongest published score, target by target (n=11n = 11), and test the gap with a two-sided Wilcoxon signed-rank test. Following the reviewer's suggestion, we add 95% bootstrap confidence intervals of each gap (20,000 resamples of the eleven targets; stratified by hub for the pooled row). We also add two further tests, a paired t-test and an exact sign-flip permutation test:

HubCompared withΔτw\Delta\tau_w95% CIWilcoxon ppt-test ppPermutation pp
Sup. CNNsITM+0.011[-0.185, +0.186]0.640.900.91
SSL CNNsPACTran+0.005[-0.025, +0.035]0.670.770.77
ViTsH-Score-0.009[-0.081, +0.070]0.700.840.84
MixedITM+0.083[-0.032, +0.193]0.240.190.20
Pooled (44 pairs)NCTI+0.102[+0.047, +0.163]0.0030.0030.002

Every per-hub interval contains zero, so on no hub does the strongest published score significantly outperform TE-Ref. The pooled interval excludes zero, and all three tests agree that TE-Ref outperforms NCTI over the four hubs. This also explains the overall advantage. The strongest published score changes from hub to hub (ITM, PACTran, H-Score and ITM), whereas TE-Ref is first on three hubs and 0.009 below the best on the fourth, and this consistency becomes a significant advantage when the four hubs are pooled.

The strongest published score is the most demanding comparison on each hub. On the supervised CNN, self-supervised CNN and mixed hubs, every published TE score has a lower mean τw\tau_w than TE-Ref. On the ViT hub, only H-Score has a higher mean, by 0.009 (p=0.70p = 0.70), and LEAD is equal to TE-Ref to three decimals.

The same reasoning applies to the pooled test. We compare against NCTI because it is the best published TE score on average over the four hubs (τw=0.567\tau_w = 0.567, Table 1), so every other published score has a lower average. Pooled over the four hubs, TE-Ref (0.6690.669) exceeds NCTI by +0.102+0.102 (p=0.003p = 0.003, two-sided Wilcoxon signed-rank test over the 44 hub-target pairs, four hubs by eleven targets). Over the three hubs that share the SFDA ground truth, TE-Ref averages 0.7010.701, against 0.6490.649 for SFDA, the best published score there (Table 2). This is the sense in which the manuscript states that TE-Ref significantly outperforms the current TE scores in the overall setting.

W4 and Q4 (Recall@1/3, selection regret and cross-fitting). We thank the reviewer for this suggestion. We computed Recall@1/3 and the selection regret of TE-Ref with the protocol of Table 33, and compare them with those of the strongest published score of each hub (Table 1). Recall@kk is the fraction of targets on which the model ranked first is among the kk best models. The selection regret is the accuracy of the best model minus that of the model ranked first, averaged over the eleven targets (in accuracy points, lower is better).

HubScoreR@1R@3Regret
Sup. CNNsTE-Ref0.730.910.085
ITM0.360.910.400
SSL CNNsTE-Ref0.550.910.411
PACTran0.550.910.411
ViTsTE-Ref0.731.000.670
H-Score0.731.000.656

TE-Ref matches or exceeds the strongest published score in Recall@1 and Recall@3 on every hub. On supervised CNNs it ranks the best model first on 8 of the 11 targets, against 4 for ITM. Its first choice there is on average 0.085 points below the best model, against 0.400 for ITM. On the self-supervised hub the two scores are tied. On ViTs, TE-Ref and H-Score have the same recall and nearly the same regret (0.670 and 0.656 points).

On cross-fitting, Table 41 (Appendix B.8) reports LDA with and without cross-fitting on the supervised CNN hub. Cross-fitting removes much of the in-sample optimism of LDA at small budgets, raising it from 0.166 to 0.531 at 10 samples per class. The effect shrinks as the budget grows: from 0.510 to 0.550 with all samples.

Inside TE-Ref this bias barely matters, because the capacity term does not depend on the target samples. Replacing LDA with its cross-fitted version changes TE-Ref by at most 0.005 at every budget, for example from 0.686 to 0.689 with all samples. Even at 10 samples per class, where LDA alone falls to 0.166, TE-Ref keeps τw=0.658\tau_w = 0.658, and 0.654 with the cross-fitted LDA.

Comments on the paper

2 comments

Dear Reviewers, we would kindly appreciate it if you could share your feedback on our rebuttal when you have a chance. Thank you for your time and consideration.

We thank all the reviewers and the meta-reviewer for engaging with our work.

The goal of our research was to provide a comprehensive evaluation of the transferability estimation (TE) literature, together with practical insights. We first aimed to identify TE scores that are reliable in a general setting: different target tasks, different model architectures and pre-training objectives, and both supervised and self-supervised CNNs and Vision Transformers. We found that none of the published scores significantly surpasses a simple reference baseline on any model hub, which we introduce as TE-Ref. Averaged over the model hubs (supervised CNNs, self-supervised CNNs, Vision Transformers, and the mixed hub), TE-Ref significantly outperforms the best published score, with a relative improvement of 18%.

We also investigated when and why these scores degrade or are not applicable, such as under label-free or data-free estimation, distribution shift in the target, and target data scarcity. In addition, the Fine-tuning Protocols paragraph raises an open question about how well TE scores can estimate fine-tuning performance when the protocol changes. We hope this work provides a basis for developing more accurate and reliable TE scores in future research. To be considered reliable and trustworthy, future scores should beat this simple, strong reference in this comprehensive setup, and should show their performance under the different situations examined in this research.

This is a summary of our rebuttal:

Clarifications: We clarified the contributions of this research: a comprehensive analysis of transferability estimation that covers where and why TE scores fail, with evaluation across different domains, models, and scenarios, and a simple reference baseline (TE-Ref) that is stronger and more reliable. Regarding the data split, we clarified that using the "all" split instead of the "train-only" split is the common approach in the literature, as in SFDA and PED. Data leakage is not relevant here, as TE scores are not trained on any data, and the "all" split is treated as a proxy of the target domain (for example, in the data scarcity experiments, a subset of the target is used). We also reported results for the "train-val" split in the rebuttal to show the impact of this choice, which is negligible based on the results provided. We again clarified that we evaluated on most of the domains and models covered in the literature, so this framework aims to be the most generalizable benchmark evaluation of the current literature, and it does not exclude any model or task with a published ground truth. Regarding significance, we clarified that although TE-Ref is not significantly better than the best TE score on any individual model hub, no TE score significantly surpasses this simple reference baseline either. Averaged over the four model hubs, TE-Ref is significantly better than any published TE score.

Insights: We discussed the fine-tuning protocols in Section 6 (Fine-tuning protocols, line 423), where we first raised this as one possible concern in the evaluation of transferability estimation. We also pointed out that this experiment uses few samples and needs further GPU-intensive investigation. In addition, we discussed other proxies of model capacity beyond the parameter count, such as the weight effective rank (which is already covered in the main draft), with more details in our response to Reviewer 9UPP.

Additional results: We provided the additional results for Table 33 on page 34, which was Reviewer gR5m's concern. We also included results that combine weight-based scores with TE scores, and results for another setup, the zero-shot evaluation of CLIP models.

Changes: The main part of the draft remains unchanged. In the revised version, we will add three appendix subsections covering the additional results discussed during the rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.