TE-Bench: A Unified Benchmark for Transferability Estimation

Transferability estimation (TE) predicts which pre-trained checkpoint will achieve the best post-fine-tuning performance on a downstream target task without fine-tuning any of the candidate checkpoints, turning a model search that would cost GPU hours to days of computation into one that runs in seconds to minutes. We built TE-BENCH, a unified evaluation of transferability estimation that covers $22$ published TE scores, several experimental setups, and a set of baselines that are rarely compared in the literature. We found that some of these baselines outperform the published TE scores, and we introduce a simple reference baseline (TE-Ref) that significantly outperforms them in the overall setting, which covers supervised and self-supervised vision models across both CNN and ViT architectures. TE-BENCH covers more than $30$ transferability estimation scores, $44$ distinct checkpoints, and $11$ target tasks.