acceptodds
Under review as a conference paper at ICLR 2027

Tailoring Transferability Estimation Evaluation to Application Preferences

Abstract

As the number of available pretrained models continues to grow, identifying the most suitable source models for a downstream task becomes increasingly difficult. Transferability estimation (TE) methods address this problem by ranking pretrained models without any exhaustive fine-tuning. Existing works evaluate these methods by measuring their agreement with a ground-truth ranking induced by a single score, mostly fine-tuned accuracy in classification. However, the accuracy may not be aligned with the objective of the downstream application, which may assign different importance to particular classes or errors, and thereby induce a different ranking of the same models. In this work, we argue that the evaluation of TE methods should explicitly account for downstream application preferences. To this end, we introduce a new framework in which ground-truth rankings are induced by performance scores tailored to the downstream application preferences. These scores form a family that includes the accuracy as a special case. When application preferences are uncertain, we evaluate TE methods across a distribution of application-conditioned ground-truth rankings. Through experiments, we show that conclusions drawn from conventional TE benchmarks do not necessarily hold across downstream preferences: ground-truth model rankings, their agreement with TE methods, as well as the induced ranking of TE methods vary with the downstream preferences. For binary classification, we further show how this variability can be visualized over the entire space of downstream preferences. The code is available in supplementary material.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
transferability estimation, transfer learning, classification, application preferences, model selection, fine-tuning, ranking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KtoAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles a problem in existing transferability estimation (TE) evaluation, which considers only accuracy and ignores the importance of each class and other metrics. The authors evaluate various pretrained models from Torchvision with various TE methods on 10 fine-tuning tasks. Also, the authors show an example of binary skin lesion classification over a grid of importance weights.

Critical strengths
  1. The paper is generally well written and easy to follow. Also, the authors clearly state the problem with the existing TE evaluation.

  2. The proposed evaluation framework is easy to adopt and can be applied universally.

Critical weaknesses
  1. The biggest concern is that the contribution is limited. The paper proposes a framework to evaluate transferability evaluation (TE) methods under non-uniform preference weights. However, the only difference from the existing evaluation is the use of importance weights. Considering the criteria of ICLR, I do not think the contribution is significant enough for acceptance. On the other hand, the paper is well written and sound. Hence, I recommend submitting this paper to another venue that puts more weight on correctness and soundness than on novelty and significance.

  2. The experiments do not match the motivating examples. The paper motivates the framework with structured preferences, such as weighting the misclassification of a malignant lesion as benign more than the confusion between two benign conditions (L50) or assigning five times more importance to outcomes involving sick patients (L196). However, the multiclass experiments never evaluate such a preference. Instead, all importance weights are sampled from uniform Dirichlet distributions (Eq. (3)). Although I appreciate that the authors conducted experiments on ISIC2019 (binary skin lesion classification), it would be better if the authors conducted experiments on more realistic scenarios.

  3. The justification for using the Dirichlet distribution is weak. The authors choose the Dirichlet distribution only because it gives a broad distribution over the simplexes of normalized weights (L260-266, L274).

  4. Although the authors show various metrics in the binary classification task, they do not show them in the multiclass classification tasks. It would be better if the authors also showed the results of various metrics in the multiclass tasks.

  5. According to L173-175, S(y^,y)S(\hat{y}, y) is defined as the unit-diagonal matrix. Then, if SISI denotes a matrix product, the numerator in Eq. (1) is identical to the denominator, which makes the ranking score always equal to 1. I suppose the authors intend element-wise multiplication between SS and II, which should be clarified.

  6. Defining aa and bb is unnecessary. Since the authors set I(tn)+I(tp)=I(fp)+I(fn)=1I(tn) + I(tp) = I(fp) + I(fn) = 1 (L219), aa and bb are simply I(tp)I(tp) and I(fn)I(fn).

  7. As the authors state in the reproducibility statement (L507-509), the results can change with the Torchvision checkpoint and the fine-tuning protocol. However, the authors do not specify how many random seeds they used in the experiments, nor do they report the standard deviations.


Minor

  1. Although readers can infer it, it would be better to define (y^,y)(\hat{y}, y) explicitly.

  2. In L227, \citet -> \citep

  3. In L219, 'Eq. (2) becomes:' should be 'Eq. (1) becomes:', since it introduces Eq. (2).

  4. It would be better if Fig. 2 appeared near L229, where it is first mentioned.

Questions for authors
  1. What is the difference between PiP_i (L170) and PP (Eq. (2)), and what does ii denote?
1 comment

Thank you for the feedback and the typos you found. We respond to your concerns below:

Limited Contribution (1) We take your concern into account. We respectfully want to highlight that our paper also investigates the impact of leveraging the score family on the evaluation of TE methods. There are other papers that also only focus on the evaluation aspect of TE methods that were published in top venues such as ICLR 2026 [1] and ECCV 2022 [2]. Also, our paper introduces the notion of distributions over importance weights, which is, to the best of our knowledge, a first in model evaluation and TE evaluation.

Experiments do not match the motivating examples and Dirichlet distributions. Indeed, sampling importances from Dirichlet distributions affect the interpretability of the importance weights. However, as noted in section 4 (paragraph on Tiles), independently rescaling the importance weights I+=(I(tn),I(tp))\mathbf{I}^+=(I(tn),I(tp)) associated with correct outcomes and the importance weights I−=(I(fp),I(fn))\mathbf{I}^-=(I(fp),I(fn)) associated with incorrect outcomes yields the same ranking of models. This scaling property allows to obtain importances that are interpretable from the sampled importances, or the opposite. As these importances induce the same rankings and that we are using rank correlation, using a II obtained by sampling is equivalent to using a specific II that would be set by hand.

We agree with you that using Dirichlet distributions may not always be the best choice. Our objective was to use a non-informative distribution to explore as many application preferences as possible and report the induced variability. Considering we can define simplexes on our distributions, we opted for the Dirichlet distribution that samples uniformly in each simplex. This is equivalent to a uniform distribution on the tile. Note that our framework can be used with any distribution a practitioner deems adequate.

Scores in mutliclass classification (4). Compared with binary classification, there are not that many current scores that are specific to multiclass classifiction. Yet, we added Table 1 to illustrate different examples.

Clarity and typos (5,6, Minors, Question 1). Thank you for highlighting where the article was lacking clarity and different typos. We have fixed them. (5) Indeed, Eq.1 is not a matrix product. We made it more clear in the paragraph above. (6) Indeed, the scores displayed on the tile are the ones that respect the condition. However, it is possible to define aa and bb without the constraint to recover other scores, such as the IoU. (Question)The difference is that PiP_i denotes the performance of the iith model while PP describes some performance without associating it to a model. This is not necessary and has been fixed.

Multiple seeds (7) Initially, the results were based on a single seed. With the computational resources that were available, we extended our experiments by fine-tuning the models on four different seeds, except DenseNet201 and ResNet152 for which we reused the original confusion matrices. We added the table in supplementary material. Conclusions still hold. Win rates still vary across TE methods and datasets, and no single TE method systematically beat the others.

Reviewer_GtQuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the problem of selecting which pretrained models, from a large set, are best aligned and therefore most likely to yield good results when fine-tuned for a specific downstream task. The authors argue that existing methods focus on ranking models by accuracy without considering the nuances of downstream tasks, where accuracy (or other classification metrics) alone is insufficient to make decisions. They propose an evaluation framework for TE methods to better understand which one works for a specific case.

Critical strengths
  • The problem is solid and important. With the explosion of new models, figuring out which model to fine-tune for downstream tasks is important, and the statement that accuracy is often not enough is also important. As more models are adapted to specific use cases, the importance of this kind of work will just increase over time.
  • The method is simple and cheap to compute, encompasses multiple views of the problem (scores), and requires no retraining. Also, the proposal for tiles make it easier, at least for binary tasks, to understand what is going on in a more visual way.
Critical weaknesses

More details on the robustness of the framework could strengthen the paper:

  • It states that a grid search for the model training was run, but does not specify which metrics were used to decide the winner. Shouldn't it be based on the downstream task's preference?
  • Some models are pretty similar in the list; it would be good to report multiple runs + std dev to see if the result holds under multiple seeds.
  • The paper talks about binary classification, but my understanding is that it assumes the threshold is 0.5 for the prediction. Could different thresholds, common on precision-focused problems like the harmful treatment examples, influence the final output?

Another issue is a lack of guidance for practitioners on how to really use this. How do we summarize the results to make a decision? How would this compare to simple baselines like just tweaking existing methods to use a weighted score instead of the average?

Questions for authors
  • Could you provide a multi-seed analysis of the results? Do they hold?
  • Could you provide a practitioner's guide on how to properly leverage your framework to make objective decisions?
  • Could we run an analysis of how the parameters of the simulation change the final output?
1 comment

\section{Response to reviewer 2}

Thank you for the great feedback. We address your concerns below:

Hyperparameter selection based on application preferences. We agree that when the application preference II is known before fine-tuning, practitionners should select hyperparameters using the corresponding validation score RIR_I. However, our experiments consider a setting in which preferences are not known before fine-tuning, as when a model provider serves downstream users with different preferences. Our fine-tuning protocol is also aligned with existing practices in TE [1]. Keeping the fine-tuning of the models fixed also allows us to isolate the effect of the scores used. Moreover, selecting hyperparameters based on importance is expensive in practice. Importance weighs are defined on spaces that grow quadratically with the number of classes and the number of rankings of hyperparmeters explodes with the size of the grid search. It is then complicated to know that there is no importance II weights for which the specific hyperparameters are the best. Thus, it is necessary to keep the model weights corresponding to every hyperparameter on each dataset, model architecture and seed.

Multiple seeds Initially, the results were based on a single seed. With the computational resources that were available, we extended our experiments by fine-tuning the models on four different seeds, except DenseNet201 and ResNet152 for which we reused the original confusion matrices. We added the table in supplementary material. Conclusions still hold. Win rates still vary across TE methods and datasets, and no single TE method systematically beat the others.

Practitioner guide Thank you for the recommendation. We have added Table 2 and Figure 2 in the main text to guide practitioners. A difficulty when dealing with uncertain importances is to define an appropriate distribution. Considering we can define simplexes on our distributions, we opted for Dirichlet distributions in our work. However, our framework can be used with any distribution the practitioner deems adequate. The important step is to select the parameters of the distributions according to the application preferences. Practitioners can specify the moments of the distribution (what is important on average, what are they uncertain abbout), use symmetries if different outcomes are equally desirable, and then leverage these constraints to identify parameters.

For summarizing the conclusions obtained when considering distributions of importances II, we use win-rates (as defined in section 6.2). The TE method with the largest win-rate, and its recommendation, are then considered the best as it mean that if we were to sample the application preferences, this TE method would be the one that is the best most of the time. This choice is motivated by prior works on stability analysis [2] and by the fact that a simple aggregation method such as averaging ranking scores lead to scores that break the mathematical properties of the performance ranking scores as defined in [1].

Binary classification threshold. Our experiments use the argmax rule, which indeed corresponds to a threshold of 0.5 in binary classification. It is also true that selecting a different threshold can change the confusion matrices and, consequently the ground-truth rankings as well as TE method evaluation results. Discussing thresholds values would greatly extends the scope of the paper, which focuses on crisp classification. However, we want to highlight that our evaluation framework can still be used if discussing thresholds. It is indeed possible to define a different model for each threshold considered and use them all to evaluate transferability estimation. This would increase memory requirements and induce many ties in the ranking established by the different transferability scores, as this would not change their input.


[1] Wenqi Shao, Xun Zhao, Yixiao Ge, Zhaoyang Zhang, Lei Yang, Xiaogang Wang, Ying Shan, and Ping Luo. Not all models are equal: Predicting model transferability in a self-challenging fisher space. In European Conference on Computer Vision (ECCV). 2022.

[2] Andrea Agostinelli, Michal P´andy, Jasper Uijlings, Thomas Mensink, and Vittorio Ferrari. How stable are transferability metrics evaluations? In European Conference on Computer Vision (ECCV). 2022.

Reviewer_beCGOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents an evaluation of TE methods and argues account for downstream application preferences and introduce a new framework in which ground-truth rankings are induced by performance scores tailored to the downstream application preferences. To validate the claims paper presents results with Tiles and benchmarks.

Critical strengths
  • The premise is interesting to transferability literature, downstream applications are valid, and this makes paper interesting to read.
Critical weaknesses
  • The paper has several critical flaws that are very hard to ignore First:
  • A big part of paper uses Tiles, which they use without citing it next to tiles, so my best guess its this work: .Sebastien Pierard, Ana ´ ¨ıs Halin, Anthony Cioppa, Adrien Deliege, and Marc Van Droogenbroeck. ` The Tile: A 2D map of ranking scores for two-class classification. arXiv, abs/2412.04309,2024.

Though interesting, the method is not adopted at all by the broader machine learning community and has only 4 citations by the authors of the original paper. This makes this method inappropriate for validation of the paper, as the original paper is also not published in a top venue. There is also an overcitation of Pierard et al. papers throughout the paper, with several important omissions in the transferability literature.

Secondly,

The experimental setup is critically flawed; the paper cites Singh(2026) in the work but uses the same setup singh et al. showed to be flawed in their paper, with static ranking and flawed scores. This does raise the question of validation for me, since the old benchmark has shown to be flawed, building another work on it does make the findings questionable here as well.

The paper lacks depth and analysis:

The work shows the results on a fairly old and empirically flawed benchmark, but this does not show broader application of what author proposes, the applications are limited to image classification. The work misses applications in regression, audio, video, time series, NLP and SSL. This makes the paper's results only applicable to a niche setup which, as I wrote earlier, is flawed as well.

Writing:

The paper presents an analysis without the implications, i know the evaluation, but the conclusion misses what was presented and key takeaways from the paper.

Questions for authors
  • Why use Tiles? why not any other method which is adopted by the community?
  • Can we use other benchmarks? Singh et al also proposed a newer benchmark, can you try that? or the benchmark from other works? such as Nguyen 2023(https://openreview.net/forum?id=1g1zg2dgIH) or Sorkhei (TMLR: k-NN as a Simple and Effective Estimator of Transferability, https://openreview.net/forum?id=hGlkjP1zHc)
  • How can the qualitative evaluation in Section 6.2 be considered "qualitative"?
1 comment

Thank you for the feedback. We address your concerns below:

Tiles citation and rationale. First, thank you for highlighting that we should cite the paper introducing tiles [1] more thoroughly in our work. We fixed it in the revision.

Why use tiles ? To the best of our knowledge, this is the only visual tool that allows to explore the variability in rankings and allow to link the different scores with application preferences. Other tools such as Precision-Recall (PR) space or Receiver Operating Characteristic (ROC) curves are not fitted as each points in PR spaces and ROC correspond to performances obtained by a specific decision thresholds on the predictions provided by a soft classifier. In comparison, each point in a tile correspond to a ranking obtained by weighting the different classification outcomes (y^,y)(\hat{y}, y) of a crisp classifier. Moreover, tiles are visual tools obtained by applying the ideas introduced at the top venue CVPR 2025 in [2] on the specific case of binary classification.

Flawed benchmark. We retained the original model hub as it is largely used in the literature [3, 4, 5, 6] and it allows to inspect previous conclusions from a new perspective. Indeed, Singh(2026)[4] did show that a static ranking outperformed many TE methods on this benchmark. However, the authors only worked with accuracies. Their observations do not hold across other scores, that may be more appropriate in other downstream applications. As this is a good argument in favour of our work, we have added a paragraph on this in section 6.1 and in the supplementary materials. Here is our findings:

A) The static ranking does not predict every application-specific ground-truth, as there is a large range of correlations between them. Despite good correlation with the accuracies, the static ranking even reached mnimal correlations of less than 0.3 in Pets, Cars, DTD and Flowers. Violin plots are available in supplementary material.

B) The static ranking does not systematically beat other TE methods when considering the diversity of ground-truth rankings. It is beaten more than 85%85\% percent of the time in Flowers, Pets and Cifar10. The table is available in supplementary material.

Extending the scope. We agree that it is interesting to extend the evaluations. It is indeed possible to perform transferability estimation methods on diverse tasks (classification, regression, ...), different data modalities (audio, video, time series, text, ...) or different hubs (models trained using different SSL methods, ...). We made the choice of aligning our work with a model hub and tasks that are mostly used in the literature [3, 4, 5, 6]. Moreover, while developing our framework in other tasks is possible, doing so requires motivating the choice of satisfaction SS and importances II, questioning how to explore different application preferences, as well as identifying how to summarize results. Therefore, we leave evaluation on other taks to future work

Conclusions of the work. Our conclusions may need to be more clearly expressed. We showed that that the conclusions in the evaluation of TE methods change with different application-specific scores that can be chosen by practitioners. Win rates vary across TE methods and datasets, and no single TE method systematically beat the others. Therefore, we hope future work will condition their ground-truth with respect to the downstream application or explore a distribution of application preferences weights II.

How can the qualitative evaluation in Section 6.2 be considered "qualitative"? We refer to it as qualitative because it is based on the scatter plots in figure 4 and does not rely on any number.

[1] Sébastien Piérard, Anaïs Halin, Anthony Cioppa, Adrien Deliège, and Marc Van Droogenbroeck. The Tile: A 2D map of ranking scores for two-class classification.

[2] Sébastien Piérard, Ana¨ıs Halin, Anthony Cioppa, Adrien Deliège, and Marc Van Droogenbroeck. Foundations of the theory of performance-based ranking. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2025

[3] Mohsen Gholami, Mohammad Akbari, Xinglu Wang, Behnam Kamranian, and Yong Zhang. ETran: Energy-based transferability estimation. In IEEE/CVF International Conference on Computer Vision (ICCV). 2023

[4] Prabhant Singh, Sibylle Hess, and Joaquin Vanschoren. How NOT to benchmark your SITE metric: Beyond static leaderboards and towards realistic evaluation. In International Conference on Learning Representations (ICLR). April 2026.

[5] Wenqi Shao, Xun Zhao, Yixiao Ge, Zhaoyang Zhang, Lei Yang, Xiaogang Wang, Ying Shan, and Ping Luo. Not all models are equal: Predicting model transferability in a self-challenging fisher space. ECCV 2022.

[6] Zixuan Hu, Xiaotong Li, Shixiang Tang, Jun Liu, Yichun Hu, and Ling-Yu Duan. LEAD: Exploring logit space evolution for model selection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) June 2024.

Comments on the paper

1 comment

We thank the reviewers beCG, GtQu, KtoA for their supportive reviews and for highlighting the interest of the paper. Transferability estimation is an important problem considering the explosion of new models (GtQu). Stating that solely relying on accuracy for the ground-truth rankings affects conclusion is important (GtQu), considering that methods keeps getting tailored to specific applications (GtQu). Our paper clearly highlights that such considerations are not considered in previous TE evaluation (KtoA). Our paper proposes an evaluation that deals with this issue by encompassing multiple views of the problem with scores tailored to applications (GtQu), that is easy to adopt (KtoA), and that can be applied universally (KtoA). As noted by reviewer GtQu, our method is also computationally cheap as no additional fine-tuning is necessary. Our paper also proposes to use tiles to visually show what is going on, which makes it easier to understand (GtQu). We are glad that beCG found our paper both interesting for the transferability literature and to read. We further appreciate that KtoA found our paper generally well-written, easy to follow and sound. Finally, we agree with GtQu that works such as this one will become more and more important over time. We have responded to the different concerns of the reviewers under their respective review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.