acceptodds
Under review as a conference paper at ICLR 2027

Understanding Cross-Task Transfer in Fine-Tuned Sentence Embeddings

Abstract

Fine-tuning a pre-trained sentence embedding model for downstream tasks reshapes its latent representation spaces, yet the relationship between these representational changes and knowledge transfer across tasks remains poorly understood. In this work, we present a systematic analysis of cross-task transfer for a syntactically informed sentence embedding model fine-tuned separately on eight MTEB classification tasks. We show that while fine-tuning consistently improves performance on its corresponding task, this comes at the cost of reduced general linguistic information in the representations, and induces both positive and negative transfer to other tasks, which is often asymmetric rather than reciprocal. To explain these behaviours, we relate transfer performance to a range of measurable properties: dataset characteristics such as training set size and vocabulary divergence, CCA-based measures of task-associated information and affinity between tasks' representational subspaces, and the geometric linearity relating different tasks' latent spaces. We found that many of these properties were strongly correlated with transfer performance improvement, most notably changes in task-associated information induced by fine-tuning. Together, these results offer an initial account of the properties associated with task transfer in fine-tuned sentence representations.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Sentence Representation Learning, Cross-task Transfer, Representation Geometry, Task Affinity, Canonical Correlation Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MA9yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors focus on the transferability between different tasks during the fine-training process of a sentence embedding model. They use different metrics to observe the transferability, including the size of dataset, the measures designed based on CCA and similarity of representations.

Critical strengths
  1. The understanding of the transferability between different tasks is quite important for providing the guidance of training language models. Currently, the understanding is still quite limited. Consequently, the analysis in this paper is valuable.
  2. The authors analyze the cross-task transfer with different metrics. The results indicate several interesting future directions to improve the understanding and prediction ability of cross-task transfer.
  3. Some empirical results are interesting. For example, Figure 1 shows strong correlations between CCA-based scores and transfer improvements.
Critical weaknesses
  1. My main concerns lie in that the current experiments are not sufficient to verify the conclusion introduced in this paper. For example, the experiments are only evaluated on a single benchmark with a specific model. It would be much better to consider additional language models with different architectures, additional benchmarks with different tasks and additional baselines.
  2. I think the presentation can be further improved. For example, if I do not misunderstand, Table 3 does not deliver additional important messages compared to Table 2. It would be better to provide some more informative Figrues and Tables, e.g., I do not see a unified comparison with different metrics in this paper.
  3. This paper evaluates the correlation between different metrics and transfer improvements and finds some useful metrics. I think it would be interesting if the authors can provide some practical applications on how can we use these metrics during the training process of language models.
Questions for authors

See Weaknesses.

1 comment

We thank the reviewer for the constructive feedback. In response, we have explicitly scoped our conclusions to the model and benchmark studied, merged Tables 2 and 3 and added a unified comparison of all six measures (new Table 3) and a heatmap of the improvements (Table 2), and clarified in the conclusion how the measures could be used in practice. Our point-by-point responses follow.

Comment 1: My main concerns lie in that the current experiments are not sufficient to verify the conclusion introduced in this paper. For example, the experiments are only evaluated on a single benchmark with a specific model. It would be much better to consider additional language models with different architectures, additional benchmarks with different tasks and additional baselines.

Response 1: We thank the reviewer for this important observation. We agree that evaluating our approach across multiple language models, architectures and benchmarks would provide stronger evidence of the generalisability of our findings. The current study focuses on a specific experimental setting, using a single language model and eight classification tasks selected from the MTEB benchmark. These tasks provide a diverse set of classification settings, allowing us to systematically investigate pairwise cross-task transfer under a controlled experimental setup. Nevertheless, our results should be interpreted within this scope, rather than as establishing that the observed relationships hold universally across different models and benchmarks.

Extending the experiments to additional models and benchmarks would be valuable, but would also require a substantial expansion of the experimental study, including retraining and re-evaluating the models and repeating the associated representational analyses. We have therefore focused the present revision on clarifying the scope of our conclusions and explicitly acknowledging the limitations concerning model and task diversity.

We have revised the conclusion section to ensure that our claims are appropriately scoped to the experimental setting investigated. In particular, we clarify that, although the selected tasks provide a diverse basis for examining cross-task transfer, the generalisability of our findings to other model architectures and task types remains to be established.

Comment 2: I think the presentation can be further improved. For example, if I do not misunderstand, Table 3 does not deliver additional important messages compared to Table 2. It would be better to provide some more informative Figrues and Tables, e.g., I do not see a unified comparison with different metrics in this paper.

Response 2: We thank the reviewer. Table 3 in the original manuscript contains the same information as Table 2, except for the base-model row, resulting in some overlap. We retained the improvement matrix because it serves as the basis for the analyses in Section 4. To reduce redundancy and improve the presentation, we have merged the two tables: the revised Table 2 includes the base-model accuracies alongside the performance improvements, while the raw-accuracy matrix (original Table 2) has been moved to the appendix.

We have also added a unified table comparing all six pairwise measures against the same transfer matrix, summarising their inputs, directionality, and corresponding Mantel correlation coefficients and p-values. Additionally, we have included a heatmap of performance improvements in the revised Table 2 to facilitate visual comparison.

Comment 3: This paper evaluates the correlation between different metrics and transfer improvements and finds some useful metrics. I think it would be interesting if the authors can provide some practical applications on how can we use these metrics during the training process of language models.

Response 3: The present study evaluates these measures as diagnostics of cross-task transfer rather than demonstrating a specific training or task-selection procedure. As noted in the revised conclusion, vocabulary divergence and affinity could be investigated for prioritising candidate adaptations, and CCA-based measures for monitoring transfer across adaptation stages or comparing checkpoints. Which measure is usable depends on what is available: vocabulary divergence needs only the datasets, affinity additionally needs labels, CCA-score change needs a source-adapted encoder and target labels, and inter-model mapping quality needs both adapted encoders, so it serves as a retrospective diagnostic. Using these measures for task selection or checkpoint selection would require separate validation against appropriate baselines, so we present these applications as directions for future investigation rather than demonstrated benefits.

Reviewer_7KQwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper the author analyse the relationship between the sentence embedding representation changes and knowledge transfer across tasks. And for this they used eight MTEB classification tasks. The findings shows that while finetuning improves performance it reduce general linguistic information. To analyse the behaviour the authors used measurable properties like training set size, vocabulary, CCA-based measures. affinity between tasks, and the geometric linearity relating different tasks latent space.

Critical strengths
  1. The paper provides a general understanding when the task adaptation helps and when the adaptation harms other tasks matters for reusing embedding models and studying interference.

  2. The authors combine linguistic probes, dataset, label associated subspaces, geometric mappings, which provides an emperical picture than transfer matrix alone.

  3. Evaluating the source-target combination gives a clearer picture than reporting a few selected examples.

Critical weaknesses
  1. The analyses shows the properties (dataset size, vocabulary, CCA-based measures. affinity between tasks) associated with transfer, but do not show why the transfer occured.

  2. Some transfer estimates are based on small testsets like the task HumEmo and HumToxic contain only 48 and 45 testset respectively.

  3. In conclusion, the sentence states that "Our findings provide a first step towards understanding adaptation dynamics in settings where a shared model is fine-tuned across multiple tasks, whether in parallel, as in multi-task learning, or in sequence, as in continual learning". However, the experiments involve independent single-task fine-tuning followed by cross-task evaluation, rather than joint training across tasks or sequential adaptation of a shared encoder. The conclusion should distinguish the demonstrated results from their potential implications for multi-task and continual learning.

Questions for authors
  1. Could the lower probing accuracy mean that linguistic information is harder for the probe to extract, rather than lost from the embeddings?

  2. It would strengthen the paper to add experiments on joint multi-task finetuning and sequential continual learning. These experiment would show whether the observed relationship also hold when the shared encoder is adapted across multiple tasks or sequential continual learning. This will provide a direct support to the statement discussed in the conclusion.

3 comments

We thank the reviewer for the helpful comments. In response, we have clarified that our analyses identify properties associated with transfer rather than its mechanisms, added a sensitivity analysis on the small-test-set tasks, HumEmo and HumToxic (Appendix D), revised the conclusion to separate what we demonstrate from its implications for multi-task and continual learning, and clarified our interpretation of the probing results. Our point-by-point responses follow.

Comment 1: The analyses shows the properties (dataset size, vocabulary, CCA-based measures. affinity between tasks) associated with transfer, but do not show why the transfer occured.

Response 1: We agree with the reviewer that our analyses identify properties associated with cross-task transfer rather than explain the underlying mechanisms of why transfer occurs. Our broader aim is to understand which characteristics of embedding spaces underlie cross-task transfer. The present study provides controlled empirical evidence about representation properties associated with transfer, rather than a causal or mechanistic explanation. We view this empirical characterisation as an initial step towards that broader goal.

Comment 2: Some transfer estimates are based on small testsets like the task HumEmo and HumToxic contain only 48 and 45 testset respectively.

Response 2: We thank the reviewer for pointing out this limitation. HumEmo and HumToxic are standard tasks included in the MTEB benchmark, but their test sets contain only 48 and 45 examples, respectively. Consequently, transfer estimates for these target tasks may be subject to greater sampling variability than those for tasks with larger test sets.

To examine the influence of these two tasks on our findings, we repeated all Mantel tests (10,000 permutation samples) after excluding HumEmo and HumToxic, leaving six tasks and 36 source--target pairs. The results show that the association between information change and transfer improvement remains strong and statistically significant (r = 0.791, p = 0.0032). The two linearity measures also retain statistically significant associations (cosine: r = 0.495, p = 0.0448; RMSE: r = -0.443, p = 0.0249). The other measures also retain the direction of their original associations, although their statistical significance is reduced in this analysis: KL divergence (r = -0.498, p = 0.2265), base affinity (r = 0.515, p = 0.5334), and impact-weighted affinity (r = 0.639, p = 0.1439).

We note that excluding these two tasks reduces the number of task-level observations from 64 to 36, which can affect the statistical power of the correlation analysis. Furthermore, with six tasks, the number of distinct task permutations available for the Mantel tests is substantially smaller (6! = 720) than with the original eight tasks (8! = 40,320). Thus, the reduced statistical significance of some associations should be interpreted in the context of this smaller evaluation setting.

We therefore consider this analysis a sensitivity analysis rather than a reason to exclude HumEmo and HumToxic from the main experiments. Retaining all eight tasks preserves the breadth of our selected benchmark and the number of observations available for the pairwise transfer analysis. We have included the additional results in the revised appendix.

Comment 3: In conclusion, the sentence states that "Our findings provide a first step towards understanding adaptation dynamics in settings where a shared model is fine-tuned across multiple tasks, whether in parallel, as in multi-task learning, or in sequence, as in continual learning". However, the experiments involve independent single-task fine-tuning followed by cross-task evaluation, rather than joint training across tasks or sequential adaptation of a shared encoder. The conclusion should distinguish the demonstrated results from their potential implications for multi-task and continual learning.

Response 3: We thank the reviewer for pointing out this distinction. We agree that our experiments investigate cross-task transfer following independent single-task fine-tuning, rather than joint multi-task training or sequential adaptation of a shared encoder. We have therefore revised the sentence in the conclusion to clarify the scope of our experimental findings while retaining their potential implications for understanding adaptation dynamics in multi-task learning and continual learning.

The revised sentence reads: “Our analysis characterises how a single round of task-specific fine-tuning affects performance on other tasks. Whether the identified relationships also hold when a shared encoder is adapted jointly across tasks (multi-task learning) or sequentially (continual learning) remains an open question and an important direction for future work.”

Question 1: Could the lower probing accuracy mean that linguistic information is harder for the probe to extract, rather than lost from the embeddings?

Response 1: We agree that a reduction in probing accuracy does not necessarily imply that the corresponding linguistic information has been erased from the embeddings. Some information may instead become encoded in a less accessible form during task-specific fine-tuning.

From a functional perspective, fine-tuning may reorganise information relevant to the target task into a more accessible form, potentially at the expense of information relevant to other tasks. This could involve either the complete loss of some information or a reduction in its accessibility, making it harder to recover or utilise.

Both scenarios can lead to lower probing accuracy and potentially affect downstream tasks that rely on the corresponding linguistic properties. Therefore, we interpret the reduction in probing accuracy as evidence of reduced accessibility of linguistic information, rather than necessarily as evidence of its complete loss.

We clarify this distinction in the revised manuscript.

Question 2: It would strengthen the paper to add experiments on joint multi-task finetuning and sequential continual learning. These experiment would show whether the observed relationship also hold when the shared encoder is adapted across multiple tasks or sequential continual learning. This will provide a direct support to the statement discussed in the conclusion.

Response 2: We thank the reviewer for this valuable suggestion. We agree that investigating joint multi-task fine-tuning and sequential continual learning would provide further insights into whether the observed relationships between task affinity and transfer also hold in these settings.

Our experiments focus on a single round of fine-tuning, where a base encoder is independently fine-tuned on each of eight tasks, allowing us to isolate how task-specific adaptation reshapes the representation space. In contrast, joint multi-task training and sequential continual learning introduce additional factors, such as shared-parameter optimisation, gradient interference, forgetting and task ordering, requiring a separate experimental design and more extensive evaluation.

While investigating these settings is beyond the scope of the current work, we have acknowledged this limitation in the revised manuscript and identified extending our analysis to joint multi-task fine-tuning and sequential continual learning as an important direction for future research.

Thank you to the authors for the detailed response and for adding the sensitivity analysis on the small-test-set tasks in Appendix D. I appreciate the clarification given for lower probing accuracy. Overall, the revision address my concerns, but given the remaining uncertainty around several transfer and the open question of what mechanisms explain them, I will retain my original rating.

Reviewer_pjhDOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates cross-task transfer behavior after fine-tuning sentence embedding models. The authors fine-tune models separately on eight MTEB classification tasks, analyze positive and negative transfer and their asymmetry, and explore factors affecting cross-task transfer performance through lexical distribution differences, CCA-based task-relevant information, subspace affinity, and linear relationships in latent spaces. The experiments show that task-specific fine-tuning generally reduces the preservation of general linguistic information, while changes in CCA-based task-relevant information exhibit a strong correlation with transfer performance.

Critical strengths

1.Practically meaningful research question. The paper systematically investigates how task-specific fine-tuning affects performance on other tasks, revealing positive and negative transfer as well as asymmetric transfer patterns. These findings contribute to understanding the transfer properties of sentence representations.

2.Systematic experimental design. The authors construct an 8×8 cross-task transfer matrix and evaluate the results using 20 random seeds, providing substantial empirical evidence for transfer relationships between tasks.

3.Comprehensive analytical perspectives. The paper examines transfer behavior from multiple perspectives, including data distributions, CCA-based task-relevant information, and latent-space geometry. In particular, changes in CCA-based information exhibit a strong correlation with transfer performance.

Critical weaknesses
  1. Potential mathematical issue in the definition of CCA-based task affinity. According to Equation (2), for a task with cc classes, there can be at most c−1c-1 nonzero canonical correlations. However, Section 4.3 uses all cc directions in UU to construct the task subspace, including directions associated with zero canonical correlations. This may cause the computed task affinity to depend on arbitrary subspace directions.
  2. Task affinity cannot explain asymmetric transfer. The task affinity measures defined in Equations (4) and (5) are both symmetric, i.e., A(i,j)=A(j,i)A(i,j)=A(j,i), whereas the transfer relationships in Table 3 are clearly directional. Therefore, these measures cannot distinguish transfer performance in opposite directions, limiting their ability to explain one of the paper's central phenomena.
  3. The main correlation finding may be influenced by the experimental design. The strong correlation (r=0.8682r=0.8682) reported in Section 4.2 includes eight self-task fine-tuning results, while HumEmo and Emotion have nearly identical lexical distributions. The paper does not report whether the correlation remains strong after excluding diagonal entries and highly similar tasks, making it difficult to determine whether this finding generalizes to cross-task transfer more broadly.
Questions for authors

See Critical Weaknesses.

2 comments

We thank the reviewer for the careful and technically detailed feedback. In response, we have added a sensitivity analysis for the zero-correlation direction in the task affinity measures (Section 4.3), added robustness analyses excluding self-task pairs, HumEmo, and both HumEmo and HumToxic (Appendix D), and clarified that our affinity measures are symmetric and cannot capture directional transfer. Our point-by-point responses follow.

Comment 1: Section 4.3 uses all (c) directions in (U) to construct the task subspace, including directions associated with zero canonical correlations. This may cause the computed task affinity to depend on arbitrary subspace directions.

Response 1: In Section 4.2, we use the root-mean-square (RMS) of the canonical correlations. In this case, we exclude the final canonical correlation because it is zero and therefore does not contribute to the numerator. However, the affinity measure proposed in Section 4.3 is based on the RMS of the cosines of the principal angles between the task subspaces, without weighting them by their corresponding canonical correlations. Consequently, we initially included all (c) CCA directions when constructing the task subspaces.

We agree that the final CCA direction, corresponding to a zero canonical correlation, may be arbitrary and that including it could introduce an element of arbitrariness into the computed affinity. To investigate this possibility, we recalculated both affinity measures after excluding the final CCA direction. The results show negligible changes in their correlations with cross-task transfer. Specifically, based on the Mantel test with 10,000 permutation samples, the normal affinity yields a correlation of r = 0.6182 (p = 0.0298), compared with the original r = 0.6053 (p = 0.0159). Similarly, the impact-weighted affinity yields r = 0.8010 (p = 0.0071), compared with the original r = 0.8016 (p = 0.0056).

These results indicate that excluding the final zero-canonical-correlation direction has little effect on the reported aggregate correlations between task affinity and cross-task transfer. This suggests that these empirical associations do not depend strongly on including that direction. Nevertheless, we agree that excluding this direction when constructing the label-associated task subspaces is a reasonable methodological choice, as it does not capture shared label-associated information. We have added this analysis to Section 4.3. As the results are essentially unchanged, we retain the original definition in the main text.

Comment 2: Task affinity cannot explain asymmetric transfer. The task affinity measures defined in Equations (4) and (5) are both symmetric, i.e., (A(i,j)=A(j,i)), whereas the transfer relationships in Table 3 are clearly directional. Therefore, these measures cannot distinguish transfer performance in opposite directions, limiting their ability to explain one of the paper's central phenomena.

Response 2: We agree with the reviewer that the proposed task affinity measures in Equations (4) and (5) are symmetric and, therefore, cannot directly capture the directional asymmetry observed in cross-task transfer. Nevertheless, these measures can help characterise the symmetric component of task relationships, providing insight into the extent to which two tasks share similar representational structures. We believe that developing a more sophisticated affinity formulation that explicitly accounts for directional relationships is an interesting direction for future research. We acknowledge this limitation and highlight it as an avenue for future work in the revised manuscript.

Comment 3: The main correlation finding may be influenced by the experimental design. The strong correlation ((r=0.8682)) reported in Section 4.2 includes eight self-task fine-tuning results, while HumEmo and Emotion have nearly identical lexical distributions. The paper does not report whether the correlation remains strong after excluding diagonal entries and highly similar tasks, making it difficult to determine whether this finding generalizes to cross-task transfer more broadly.

Response 3: We thank the reviewer for raising this important point. We conducted additional Mantel tests (10,000 permutation samples) to examine whether our correlation findings are influenced by self-task fine-tuning results or by the inclusion of HumEmo, whose lexical distribution is highly similar to that of Emotion.

Excluding self-task fine-tuning results (56 off-diagonal pairs): The correlations for base affinity, impact-weighted affinity and information change remain strong. In fact, the correlations for base affinity and impact-weighted affinity increase from 0.605 to 0.744 and from 0.802 to 0.814, respectively. The association with information change also remains strong (r = 0.818), while the negative association with KL-divergence changes only marginally (from r= -0.589 to r=-0.562). The associations involving the linearity measures become weaker after excluding self-task results: the correlation with cosine similarity decreases from 0.398 to 0.285, while that with RMSE changes from -0.353 to -0.301. Overall, excluding self-task comparisons has little effect on the associations involving affinity and information change, while the associations involving linearity measures become weaker, suggesting that self-task results contribute to the observed relationships to varying degrees.

Excluding HumEmo (49 task pairs): We also repeated the analysis after removing HumEmo from the task set. The correlations for information change (r = 0.766, p = 0.0005), impact-weighted affinity (r = 0.672, p = 0.0303), KL divergence (r = -0.569, p = 0.0363), and both linearity measures (cosine similarity: r = 0.540, p = 0.0038; RMSE: r = -0.448, p = 0.0026) retain their original directions and show relatively strong associations. The correlation for base affinity remains positive (r = 0.493), although its associated p-value increases to 0.3370. Furthermore, after excluding both HumEmo and HumToxic (36 task pairs), the associations involving information change and both linearity measures retain their original direction, while the other measures also retain their original directions, although their correlation magnitudes and associated p-values vary.

We do not consider HumEmo and Emotion to be an exceptional pair that should necessarily be excluded from the main analysis. Their lexical similarity is itself relevant to our investigation, as one of our research questions concerns the relationship between dataset characteristics and cross-task transfer. Nevertheless, the additional analyses allow us to examine whether the observed relationships depend on the inclusion of these tasks.

Overall, these sensitivity analyses provide further support for the robustness of the relationships involving information change and impact-weighted affinity, which remain strong across the different experimental settings. The results for base affinity, KL divergence and linearity show some sensitivity to the exclusions, although their correlations generally retain their original directions. We have included these additional analyses in the revised appendix and revised the discussion to reflect these observations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.