acceptodds
Under review as a conference paper at ICLR 2027

Identification of Task Affinity for Multi-Task Learning based on Divergence of Tabular Task Data

Abstract

Multi-task learning (MTL) can significantly improve performance by training shared models for related tasks. However, due to the risk of negative transfer between mismatched tasks, the effectiveness of MTL hinges on identifying which tasks should be learned together. In this paper, we show that for tabular datasets, this affinity between a pair of tasks can be predicted based on static features that characterize the relationship between the datasets of these tasks. Specifically, we show that we can train a regression model for predicting pairwise task affinity based on computationally-efficient features, requiring ground-truth affinity values for only a small, random sample of task pairs to generalize across all possible pairs. We demonstrate on three benchmark tabular datasets that our proposed approach can predict affinity more accurately at a lower computational cost than existing methods for identifying task affinity, which treat task data as black boxes and require training-based signals. Our work provides a practical and scalable solution to task grouping for MTL, enabling its effective application to tabular datasets with large numbers of tasks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Machine Learning, Multi-Task Learning, Task Affinity, AutoML

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5GpdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an approach to predicting task affinity for multi-task learning on tabular data. It uses dataset statistics and distributional differences to train a model on a subset of observed pairwise gains. The predicted affinities then guide task grouping. Experiments evaluate prediction accuracy, computational cost, and downstream performance.

Critical strengths
  1. The paper addresses a practical problem: identifying beneficial task groups without training models on all task pairs.
  2. The proposed approach is simple and provides interpretable predictions that can be integrated into downstream tasks.
Critical weaknesses
  1. The manuscript contains many errors. For example, “conflicting gradients updates” (page 5, line 235), “to large number of tasks” (page 8, line 415), “all benchmarks datasets” (page 9, line 444), “21th” (page 11, line547), “Summery” (page 13, line 678), and “IEEE, IEEE” (page 11, line 580). The authors should revise the manuscript. Figure 1 is poorly presented, several y-axis labels are overlap. This making this figure difficult to read.

  2. The main experiments cover only three datasets and two baselines. More diverse datasets and competitive baselines are need to prove the effectiveness of the proposed method.

Questions for authors
  1. The hypothesis in Section 3 needs stronger justification. The same inputs can support different prediction tasks. For example, the same housing records could be used to predict rent and electricity consumption. But these tasks may depend on different features, which may not benefit from joint learning. Why should similar input distributions indicate positive transfer? Please clarify the reason of this.

  2. Tables 1 and 8 appear to report the same experimental setting, but their correlation values differ. The proposed method’s Chemical result is $0.50\pm0.03$ versus 0.52±0.020.52\pm0.02, while GRAD-TAE’s Landmine result is −0.26±0.00-0.26\pm0.00 versus −0.23±0.01-0.23\pm0.01. Table 4 reports Landmine’s within-task distance as 1.13±0.091.13\pm0.09, whereas the text gives 1.13±0.0091.13\pm0.009. Could the authors explain these difference?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 5Gpd

W1 & Q2: Presentation & Discrepancies

We thank the reviewer for the careful reading and for catching these typos. We have corrected all the listed errors and proofread the full manuscript. We have also redesigned Figure 1 so the y-axis labels no longer overlap.

We apologize for the inconsistencies. The values in Table 8 (Chemical correlation and Landmine GRAD-TAE) were typos. Table 1 and the main text report the correct values (Chemical: +0.50 correlation; GRAD-TAE on Landmine: −0.26). We have updated Table 8 and the corresponding text in Appendix A.4. We have also corrected the Landmine within-task distance in the Appendix text (1.13 ± 0.09) so that it matches Table 4.

W2: Datasets and Baselines

We agree that a broader evaluation would strengthen the paper. However, conducting additional experiments is not feasible within the rebuttal period. We note the following regarding the scope of our evaluation: School, Chemical, and Landmine are standard tabular MTL benchmarks (Zhang & Yang, 2021). They cover both regression and classification and span different levels of inter-task divergence (Table 4). Our comparison includes three baselines: TAG and GRAD-TAE in the main text, and the Linear Surrogate model (Li et al., 2023) in Appendix A.5. Other task-grouping methods, such as Higher-Order Approximation (HOA; Standley et al., 2020) and the meta-learning-based predictor MTGNet (Song et al., 2022), require exhaustive or extensive MTL training. HOA is equivalent to the exhaustive pairwise evaluation that we use for ground truth.

Q1: Justification of the Hypothesis

Our setting assumes that all tasks share a common input schema, with each task having its own dataset (e.g., different schools, molecules, or landmine fields). We agree that similar input distributions alone do not guarantee positive transfer. For example, two tasks could have similar input distributions but substantially different input-to-label relationships, potentially leading to negative transfer. Our hypothesis is therefore not that input similarity guarantees positive transfer, but that input-distribution similarity can provide a useful signal of task relatedness.

In our setting, the hypothesis is motivated by two considerations: Theory: domain-adaptation theory (Ben-David & Borbely, 2008) bounds cross-task generalization error by the divergence between input distributions. Energy Distance and Feature-Mean Gap serve as efficient proxies for this divergence. Empirical support: these features correlate negatively with MTL gain (Appendix A.2.2), consistently in Landmine and more weakly in School, where all correlations are below 0.1 in magnitude. We therefore treat the hypothesis as a useful prior, not a universal rule. So, we learn a regression model on observed gains rather than using a single similarity score.

Reviewer_LEG3OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors study the estimation of task affinity in the context of multi-task learning (MTL). MTL can suffer from negative transfer when tasks that are not sufficiently compatible are trained jointly. The authors propose learning a regression function that estimates task affinity using a limited set of observed ground-truth task affinities. They investigate a range of features describing the datasets associated with a given pair of tasks, including relative dataset sizes, distributional distances, PCA-based similarities, and cross-task edges in a graph constructed from the union of the datasets. These features are used to predict task affinity through regression. The authors analyze their predictive performance and evaluate the resulting impact on downstream task grouping.

Critical strengths
  • Determining task relationships for MTL is an important problem that remains challenging. Being able to identify potentially beneficial task combinations before performing extensive joint training is practically relevant, although the proposed method still requires some observed MTL gains for supervision.
  • The proposed approach extracts features directly from the datasets using relatively simple statistical and structural measures. This is conceptually straightforward and potentially applicable to a variety of settings.
  • The paper is well written and generally easy to follow.
Critical weaknesses
  • Technical novelty appears somewhat limited. This is not necessarily a serious limitation, but it places greater importance on demonstrating practical significance. In this respect, the downstream evaluation could be strengthened. The experiments are conducted on a limited collection of established MTL benchmarks, and it would be useful to investigate more challenging settings. The authors should also clarify the definition of 'total loss' in the downstream evaluation, including whether it is computed on training or test data and how individual task losses are aggregated. Also, an extensive analysis of how improved task-affinity prediction translates into better overall and individual task performance is required.

  • Some of the proposed features, e.g., cross-task graph connections, have an intuitive interpretation in terms of task relationships. However, the roles of other features, particularly relative dataset size, are less clear. For example, two tasks may have essentially identical underlying data distributions but substantially different sample sizes. Their statistical similarity would remain high, while their relative dataset-size features could differ considerably. Please explain how such features contribute to predicting task affinity.

  • It is not entirely clear which aspects of the proposed feature construction are specific to tabular datasets. Many of the considered features and similarity measures appear applicable to other data modalities. It would be useful to clarify whether the emphasis on tabular data reflects particular methodological requirements or primarily the chosen experimental setting.

  • The proposed approach requires ground-truth MTL gains for a subset of task pairs. This is not necessarily a limitation of the method itself, but it defines a supervised operational setting. It would be useful to discuss more explicitly the computational cost of obtaining these supervision signals and the conditions under which the proposed approach offers a practical advantage over directly evaluating task combinations.

Questions for authors

See Weaknesses section

1 comment
AuthorsOpenReview ↗

Response to Reviewer LEG3

W1: Novelty, Downstream Evaluation, and “Total Loss”

We agree that the contribution is empirical rather than a new learning algorithm. Our claim is that static, training-free dataset features can predict pairwise MTL gain better than gradient- and training-based signals (TAG, GRAD-TAE), at lower or comparable runtime (Table 1, Figure 2, and Table 8). We then demonstrate that these predictions are usable downstream with beam search (Figure 3) and SDP clustering (Table 2).

“Total loss” in Table 2 is the sum of per-task losses (MSE for School, BCE for Chemical and Landmine) over all tasks (139 for School, 35 for Chemical, 29 for Landmine) under the selected grouping (using predictions from different methods), compared against the sum of all single-task (STL) losses. It is computed on held-out test data from each task. We have added this definition to Section 4.3.3.

Per-task and overall effects are shown in Figure 3 (average per-task MTL gain of the groups selected with each method’s predictions) and Table 2 (total loss). Groups selected with our predictions attain the lowest total loss on School (130.19 vs. 131.47 STL) and Landmine (5.70 vs. 6.09 STL), and Figure 3 shows higher average per-task MTL gain with beam search on all three benchmarks. The exception is Chemical, where GRAD-TAE reaches a lower total loss (20.05 vs. our 20.73, about one standard deviation of GRAD-TAE’s result). Our method still improves on STL (22.98) and TAG (21.64) there, using k=2k=2 groups versus GRAD-TAE’s 15.

W2: Role of Dataset-Size Features

MTL gain does not depend only on distributional similarity. It is directional (Eq. 3) and depends on how much a task benefits from the other task’s data under a shared encoder. A small task paired with a large one can gain from the large amount of extra data, while a large task has less to gain. This effect motivates our asymmetric size-ratio features.

Because we equalize dataset sizes during training by repeating samples (Appendix A.1.1), the small task gets no new information from its own repeats, so a larger partner may matter more. Empirically, size features are among the strongest single predictors on Chemical and Landmine (Figure 1 in Section 4.2 and Figure 5 in Appendix A.2.2), though weak on School. We do not claim that these features measure task relatedness. Rather, they complement the distributional-divergence features, and the learned regressor combines both. We will clarify this distinction in Section 3.1.

W3: What is Specific to Tabular Data?

The features are not inherently limited to tabular data. However, tabular data provide a fixed schema with consistent feature semantics across tasks, allowing input-space distances, mean gaps, and rank divergence to be computed directly on raw inputs without pretrained embeddings or additional training. This keeps the feature computation lightweight. For images or text, the same features would likely need learned embeddings, adding computational cost and dependence on the embedding model. In many image MTL settings (e.g., object detection, scene understanding), all tasks share the same input images and differ only in their labels. In such cases, input-space divergence between tasks is zero by construction and carries no signal. Our features target settings where each task has its own dataset over a common schema, as in School, Chemical, and Landmine. Tabular data is a natural setting for this formulation and is also prevalent in many industry applications, which motivates our focus on it.

W4: Cost of Supervised Setting

We agree that our proposed approach is a supervised predictor. The reported runtimes include the cost of training MTL models on the labeled pairs (Figure 2, Table 8). With 25% of pairs, this is 4× fewer pairwise trainings than exhaustive evaluation. Figures 6 and 7 show that 5–10% of pairs already give meaningful correlation (about 0.10 on School, 0.45 on Chemical, 0.48 on Landmine), with performance saturating around 30–40%. The practical advantage is therefore greatest when (i) the number of tasks is large, since the number of pairs grows quadratically (School has 9,591 pairs), (ii) each MTL training is expensive, and (iii) pairwise predictions are reused to score larger groups (Eq. 15). With fewer tasks, directly evaluating all pairs may be simpler and more reliable. We have added this discussion to the revised manuscript (Section 4.3.2).

Reviewer_hSoiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper predicts pairwise multi-task learning (MTL) gains for tabular tasks from static task-relation features, avoiding exhaustive joint training over all task pairs. Features capture dataset size, input-space distances, distributional divergence, representation similarity, and KNN-graph overlap. A regularized quadratic regression model is trained on ground-truth gains for a small subset of pairs and generalizes to unseen pairs. On School, Chemical, and Landmine, it achieves higher correlation with true MTL gains and lower runtime than TAG and GRAD-TAE, and supports downstream beam-search and SDP-based task grouping.

Critical strengths

The problem is practically relevant and clearly motivated: replacing repeated MTL training with cheap, interpretable dataset statistics could substantially lower the cost of task grouping. The empirical evidence is consequential. The experimental results appear promising. The feature-level analysis and data-efficiency curves further show that meaningful prediction is possible from a small fraction of labeled pairs.

Critical weaknesses

The primary weakness is insufficient methodological novelty. The contribution is essentially a broader hand-crafted feature set combined with standard quadratic ridge regression. Prior work already studies task-affinity estimation and task-characteristic-based transferability, so the advance is incremental and primarily empirical.

Generality is also limited: features assume tabular inputs with shared dimensions, and all validation uses three tabular benchmarks. In addition, feature selection and cross-validation grouping are under-specified, raising concerns about optimistic evaluation.

Questions for authors

(1) What is the precise conceptual or technical advance beyond existing task-affinity and task-characteristic methods, rather than a broader feature set?

(2) Were feature selection and correlation analysis restricted to training pairs, and were both directions of each unordered pair kept in the same CV fold?

(3) Since labeling 25% of pairs still scales quadratically, please distinguish constant-factor savings from improved scaling order.

(4) School's Table 2 losses, 130.19 +/- 0.21 versus TAG's 130.30 +/- 0.20, do not alone establish a significant difference.

2 comments
AuthorsOpenReview ↗

Response to Reviewer hSoi

Q1 & W1: Novelty

We thank the reviewer for the thoughtful assessment. We agree that our contribution is primarily empirical and that the regression model itself is standard. The novelty lies in the formulation: pairwise MTL gain is predicted from static, training-free features of the two tasks’ datasets, using ground-truth gains for only a small subset of pairs. Prior data-driven methods (e.g., Linear Surrogate, GRAD-TAE, MTGNet) require many MTL training runs, and TAG requires gradient probing during a large-scale MTL training. Prior characteristic-based work (Bingel & Søgaard, 2017; Ben-David & Borbely, 2008) identifies properties correlated with transfer, but does not predict pairwise MTL gain or support task grouping. Our results show that this lightweight formulation predicts gain better than TAG and GRAD-TAE at comparable or lower runtime (Table 1, Figure 2), and that it can drive beam search and SDP grouping (Figure 3, Table 2), which is important for practical applications. We will state the scope of our contribution more precisely in the introduction.

W2: Generality

We acknowledge that our evaluation focuses on three tabular benchmarks. However, tabular data remain the primary modality in many industry applications, including finance and recommendation systems. Our features are purposefully designed to exploit the fixed-schema nature of tabular data, enabling lightweight and interpretable task-relation measurements without expensive pretraining or embedding extraction.

In principle, our framework could also be extended to unstructured modalities (e.g., images) using appropriate feature extractors or learned embeddings, but designing and evaluating such extensions is beyond the scope of this work and is left for future research.

Q2: Feature Selection and Cross-Validation

Feature selection and the correlation analysis (Figure 1 in Section 4.2 and Figure 5 in Appendix A.2.2) use only the training pairs. The train/test split and the cross-validation folds are formed at the level of unordered pairs, so both directional labels of a pair always stay together. This prevents information leakage between training and validation/test sets and ensures a fair evaluation without optimistic bias. We have stated this explicitly in Sections 3.2 and 4.1 of the revised manuscript.

Q3: Scaling

Our approach does not require exhaustive pairwise evaluation. Meaningful predictive performance can be achieved with only a fraction of the pairs: even 5–10% of pairs yield meaningful correlations with observed MTL gains (Figure 6), while the remaining steps (feature computation, regression fitting, and inference) incur negligible cost (Appendix A.2.1).

The dominant cost is collecting ground-truth MTL gains, which depends on the number of labeled pairs and the cost of each pairwise MTL training. Importantly, the required number of labeled pairs depends primarily on the difficulty of the affinity prediction problem and the complexity of the predictor (Figure 6, Appendix A.4), rather than the total number of possible pairs. While the required number of pairs may increase with the number and diversity of tasks, we expect it to scale sub-quadratically when tasks and their relationships are drawn from the same distribution. We have clarified this distinction in the revised manuscript (Section 4.3.2).

Q4: Significance on School

We agree that on the School dataset, the difference between our total loss (130.19 ± 0.21) and TAG’s (130.30 ± 0.20) is within run-to-run variation, and we will not claim an improvement over TAG in terms of total loss in this specific case.

We note, however, that TAG’s affinity estimation takes 89 minutes on School versus 18 minutes for our approach (Figure 2, Table 8), and its affinities are nearly uncorrelated with the true gains (0.002 for TAG vs. 0.13 for ours). In short, we match TAG’s downstream performance at about a fifth of the runtime. Our groups also improve on STL (131.47) and on GRAD-TAE (130.74 ± 0.30).

School is also the hardest benchmark (Table 4: highest input divergence; Figure 5: weak feature correlations throughout), so small margins are expected for every method. The result is consistent with the other benchmarks: our groups improve on STL on all three, and give the lowest total loss on Landmine (5.70 vs. 6.09 for STL, 6.15 for GRAD-TAE, and 6.25 for TAG).

Thank you for the responses that had addressed most of my concerns. Although the method is applicable to various industry applications, I still feel that the methodological novelty is somewhat limited. Therefore I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.