Good Sources or Good Matches: Disentangling Source Quality in Language Model Transfer

Data-selection methods seek to predict which training sources will benefit a target, but prediction accuracy alone does not reveal whether they identify generally useful sources or target-specific matches. This distinction determines whether past transfer experiments can guide future selection or whether new target-specific evidence is needed. We investigate this question through adapter transfer on 15 datasets and four language models. We measure relative source usefulness through target-conditioned rankings and use HodgeRank to identify how much transfer asymmetry is explained by a common source ordering. This ordering corresponds to average source quality and explains most squared asymmetry in the decoder-only models studied; T5 shows more directional variation that a single ordering cannot explain. We then evaluate two ways to reuse measured transfer. Historical source quality guides selection for incoming targets from an established model library, outperforming the tested gradient references in the decoder-only settings. Transfer measured on a smaller pilot also supports source selection for Qwen3-8B and LLaMA, while target-model gradients remain stronger for T5. Further analysis shows that the pilot’s general source ordering predicts target-model asymmetry more accurately than its full directional structure. These results clarify what makes transfer evidence useful and provide practical ways to select source datasets using observations from previous targets or a smaller model.