- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper tackles a problem in existing transferability estimation (TE) evaluation, which considers only accuracy and ignores the importance of each class and other metrics. The authors evaluate various pretrained models from Torchvision with various TE methods on 10 fine-tuning tasks. Also, the authors show an example of binary skin lesion classification over a grid of importance weights.
Critical strengths
-
The paper is generally well written and easy to follow. Also, the authors clearly state the problem with the existing TE evaluation.
-
The proposed evaluation framework is easy to adopt and can be applied universally.
Critical weaknesses
-
The biggest concern is that the contribution is limited. The paper proposes a framework to evaluate transferability evaluation (TE) methods under non-uniform preference weights. However, the only difference from the existing evaluation is the use of importance weights. Considering the criteria of ICLR, I do not think the contribution is significant enough for acceptance. On the other hand, the paper is well written and sound. Hence, I recommend submitting this paper to another venue that puts more weight on correctness and soundness than on novelty and significance.
-
The experiments do not match the motivating examples. The paper motivates the framework with structured preferences, such as weighting the misclassification of a malignant lesion as benign more than the confusion between two benign conditions (L50) or assigning five times more importance to outcomes involving sick patients (L196). However, the multiclass experiments never evaluate such a preference. Instead, all importance weights are sampled from uniform Dirichlet distributions (Eq. (3)). Although I appreciate that the authors conducted experiments on ISIC2019 (binary skin lesion classification), it would be better if the authors conducted experiments on more realistic scenarios.
-
The justification for using the Dirichlet distribution is weak. The authors choose the Dirichlet distribution only because it gives a broad distribution over the simplexes of normalized weights (L260-266, L274).
-
Although the authors show various metrics in the binary classification task, they do not show them in the multiclass classification tasks. It would be better if the authors also showed the results of various metrics in the multiclass tasks.
-
According to L173-175, is defined as the unit-diagonal matrix. Then, if denotes a matrix product, the numerator in Eq. (1) is identical to the denominator, which makes the ranking score always equal to 1. I suppose the authors intend element-wise multiplication between and , which should be clarified.
-
Defining and is unnecessary. Since the authors set (L219), and are simply and .
-
As the authors state in the reproducibility statement (L507-509), the results can change with the Torchvision checkpoint and the fine-tuning protocol. However, the authors do not specify how many random seeds they used in the experiments, nor do they report the standard deviations.
Minor
-
Although readers can infer it, it would be better to define explicitly.
-
In L227, \citet -> \citep
-
In L219, 'Eq. (2) becomes:' should be 'Eq. (1) becomes:', since it introduces Eq. (2).
-
It would be better if Fig. 2 appeared near L229, where it is first mentioned.
Questions for authors
- What is the difference between (L170) and (Eq. (2)), and what does denote?
Thank you for the feedback and the typos you found. We respond to your concerns below:
Limited Contribution (1) We take your concern into account. We respectfully want to highlight that our paper also investigates the impact of leveraging the score family on the evaluation of TE methods. There are other papers that also only focus on the evaluation aspect of TE methods that were published in top venues such as ICLR 2026 [1] and ECCV 2022 [2]. Also, our paper introduces the notion of distributions over importance weights, which is, to the best of our knowledge, a first in model evaluation and TE evaluation.
Experiments do not match the motivating examples and Dirichlet distributions. Indeed, sampling importances from Dirichlet distributions affect the interpretability of the importance weights. However, as noted in section 4 (paragraph on Tiles), independently rescaling the importance weights I+=(I(tn),I(tp)) associated with correct outcomes and the importance weights I−=(I(fp),I(fn)) associated with incorrect outcomes yields the same ranking of models. This scaling property allows to obtain importances that are interpretable from the sampled importances, or the opposite. As these importances induce the same rankings and that we are using rank correlation, using a I obtained by sampling is equivalent to using a specific I that would be set by hand.
We agree with you that using Dirichlet distributions may not always be the best choice. Our objective was to use a non-informative distribution to explore as many application preferences as possible and report the induced variability. Considering we can define simplexes on our distributions, we opted for the Dirichlet distribution that samples uniformly in each simplex. This is equivalent to a uniform distribution on the tile. Note that our framework can be used with any distribution a practitioner deems adequate.
Scores in mutliclass classification (4). Compared with binary classification, there are not that many current scores that are specific to multiclass classifiction. Yet, we added Table 1 to illustrate different examples.
Clarity and typos (5,6, Minors, Question 1). Thank you for highlighting where the article was lacking clarity and different typos. We have fixed them. (5) Indeed, Eq.1 is not a matrix product. We made it more clear in the paragraph above. (6) Indeed, the scores displayed on the tile are the ones that respect the condition. However, it is possible to define a and b without the constraint to recover other scores, such as the IoU. (Question)The difference is that Pi denotes the performance of the ith model while P describes some performance without associating it to a model. This is not necessary and has been fixed.
Multiple seeds (7) Initially, the results were based on a single seed. With the computational resources that were available, we extended our experiments by fine-tuning the models on four different seeds, except DenseNet201 and ResNet152 for which we reused the original confusion matrices. We added the table in supplementary material. Conclusions still hold. Win rates still vary across TE methods and datasets, and no single TE method systematically beat the others.