BOTAS: Bayesian Optimization for Zero-Shot Source Task Selection in Merging of LLMs

Model merging combines independently fine-tuned checkpoints into a single model without joint training. However, existing evaluations assess the merged model only on the same in-domain source tasks, leaving zero-shot transfer to an unseen target task underresearched. To this end, we conduct an exhaustive study of source task subset selection for model merging, evaluating all $4,083$ subsets of 12 source tasks against 12 disjoint held-out target tasks, each at 40 scaling coefficients. Our results show that transfer is strongly aligned with the task family and that optimal mixtures typically involve fewer than 5 source tasks. We also show that a naive training-free solution based on training-free geometric selection, i.e., a task similarity measured by alignment between task vectors, does not identify the best mixture. To address this, we introduce BOTAS (Bayesian Optimization for TAsk Selection), which models the discrete space of source subsets with a Gaussian process surrogate. Without access to any target training data, BOTAS recovers the true optimal subset in 83% of runs while exploring only 1.5% of the search space. In addition, we show that in four out of twelve target tasks, the best merged subset outperforms full fine-tuning on the target itself. Finally, we release the full evaluation as a reusable benchmark for future source-selection methods.