Understanding Cross-Task Transfer in Fine-Tuned Sentence Embeddings

Fine-tuning a pre-trained sentence embedding model for downstream tasks reshapes its latent representation spaces, yet the relationship between these representational changes and knowledge transfer across tasks remains poorly understood. In this work, we present a systematic analysis of cross-task transfer for a syntactically informed sentence embedding model fine-tuned separately on eight MTEB classification tasks. We show that while fine-tuning consistently improves performance on its corresponding task, this comes at the cost of reduced general linguistic information in the representations, and induces both positive and negative transfer to other tasks, which is often asymmetric rather than reciprocal. To explain these behaviours, we relate transfer performance to a range of measurable properties: dataset characteristics such as training set size and vocabulary divergence, CCA-based measures of task-associated information and affinity between tasks' representational subspaces, and the geometric linearity relating different tasks' latent spaces. We found that many of these properties were strongly correlated with transfer performance improvement, most notably changes in task-associated information induced by fine-tuning. Together, these results offer an initial account of the properties associated with task transfer in fine-tuned sentence representations.