- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors focus on the transferability between different tasks during the fine-training process of a sentence embedding model. They use different metrics to observe the transferability, including the size of dataset, the measures designed based on CCA and similarity of representations.
Critical strengths
- The understanding of the transferability between different tasks is quite important for providing the guidance of training language models. Currently, the understanding is still quite limited. Consequently, the analysis in this paper is valuable.
- The authors analyze the cross-task transfer with different metrics. The results indicate several interesting future directions to improve the understanding and prediction ability of cross-task transfer.
- Some empirical results are interesting. For example, Figure 1 shows strong correlations between CCA-based scores and transfer improvements.
Critical weaknesses
- My main concerns lie in that the current experiments are not sufficient to verify the conclusion introduced in this paper. For example, the experiments are only evaluated on a single benchmark with a specific model. It would be much better to consider additional language models with different architectures, additional benchmarks with different tasks and additional baselines.
- I think the presentation can be further improved. For example, if I do not misunderstand, Table 3 does not deliver additional important messages compared to Table 2. It would be better to provide some more informative Figrues and Tables, e.g., I do not see a unified comparison with different metrics in this paper.
- This paper evaluates the correlation between different metrics and transfer improvements and finds some useful metrics. I think it would be interesting if the authors can provide some practical applications on how can we use these metrics during the training process of language models.
Questions for authors
See Weaknesses.
We thank the reviewer for the constructive feedback. In response, we have explicitly scoped our conclusions to the model and benchmark studied, merged Tables 2 and 3 and added a unified comparison of all six measures (new Table 3) and a heatmap of the improvements (Table 2), and clarified in the conclusion how the measures could be used in practice. Our point-by-point responses follow.
Comment 1: My main concerns lie in that the current experiments are not sufficient to verify the conclusion introduced in this paper. For example, the experiments are only evaluated on a single benchmark with a specific model. It would be much better to consider additional language models with different architectures, additional benchmarks with different tasks and additional baselines.
Response 1: We thank the reviewer for this important observation. We agree that evaluating our approach across multiple language models, architectures and benchmarks would provide stronger evidence of the generalisability of our findings. The current study focuses on a specific experimental setting, using a single language model and eight classification tasks selected from the MTEB benchmark. These tasks provide a diverse set of classification settings, allowing us to systematically investigate pairwise cross-task transfer under a controlled experimental setup. Nevertheless, our results should be interpreted within this scope, rather than as establishing that the observed relationships hold universally across different models and benchmarks.
Extending the experiments to additional models and benchmarks would be valuable, but would also require a substantial expansion of the experimental study, including retraining and re-evaluating the models and repeating the associated representational analyses. We have therefore focused the present revision on clarifying the scope of our conclusions and explicitly acknowledging the limitations concerning model and task diversity.
We have revised the conclusion section to ensure that our claims are appropriately scoped to the experimental setting investigated. In particular, we clarify that, although the selected tasks provide a diverse basis for examining cross-task transfer, the generalisability of our findings to other model architectures and task types remains to be established.
Comment 2: I think the presentation can be further improved. For example, if I do not misunderstand, Table 3 does not deliver additional important messages compared to Table 2. It would be better to provide some more informative Figrues and Tables, e.g., I do not see a unified comparison with different metrics in this paper.
Response 2: We thank the reviewer. Table 3 in the original manuscript contains the same information as Table 2, except for the base-model row, resulting in some overlap. We retained the improvement matrix because it serves as the basis for the analyses in Section 4. To reduce redundancy and improve the presentation, we have merged the two tables: the revised Table 2 includes the base-model accuracies alongside the performance improvements, while the raw-accuracy matrix (original Table 2) has been moved to the appendix.
We have also added a unified table comparing all six pairwise measures against the same transfer matrix, summarising their inputs, directionality, and corresponding Mantel correlation coefficients and p-values. Additionally, we have included a heatmap of performance improvements in the revised Table 2 to facilitate visual comparison.
Comment 3: This paper evaluates the correlation between different metrics and transfer improvements and finds some useful metrics. I think it would be interesting if the authors can provide some practical applications on how can we use these metrics during the training process of language models.
Response 3: The present study evaluates these measures as diagnostics of cross-task transfer rather than demonstrating a specific training or task-selection procedure. As noted in the revised conclusion, vocabulary divergence and affinity could be investigated for prioritising candidate adaptations, and CCA-based measures for monitoring transfer across adaptation stages or comparing checkpoints. Which measure is usable depends on what is available: vocabulary divergence needs only the datasets, affinity additionally needs labels, CCA-score change needs a source-adapted encoder and target labels, and inter-model mapping quality needs both adapted encoders, so it serves as a retrospective diagnostic. Using these measures for task selection or checkpoint selection would require separate validation against appropriate baselines, so we present these applications as directions for future investigation rather than demonstrated benefits.