acceptodds
Under review as a conference paper at ICLR 2027

WHEN DOES TASK AFFINITY JUSTIFY PAIRWISE MULTI-TASK CONFIGURATION SELECTION?

Abstract

Task-affinity scores in multi-task learning are typically validated by their correlation with multi-task gain and then used to decide which tasks share what. We argue that these are different claims: a score (diagnostic) can track a real mechanism, such as representation reuse or output coupling, without determining the right configuration decision. We decompose this inference into three links: score → intervention effect → preferred configuration → out-of-sample selection gain, and test them on 1914 ordered pairs from 172 tasks spanning CelebA, QM9, and 16 multi-output regression families. The first link holds selectively: a direct reuse probe tracks frozen representation reuse and residual dependence tracks output coupling, but no diagnostic reliably predicts the value of adaptation. The second is context-dependent: with the measured task relation nearly unchanged, replacing a small CNN with ResNet-18 reduces adapted-transfer selection from 30% to 7%, while preferences also shift with the candidate set, metric, and target-data budget. A fixed portfolio of two or three configurations reduces oracle headroom below replicate noise on CelebA and QM9. Finally, learned selectors that beat the best fixed configuration on held-out pairs generalize to held-out tasks only on CelebA under the log score, while family extrapolation remains unsupported. We distill these results into principles for more reliable task-affinity evaluation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multi-task learning, Task affinity, Model selection, Task grouping, Transfer learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6uDVOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Task-affinity scores are normally validated by correlating them with observed multi-task gain. The authors argue that task affinity justifies pairwise MTL configuration selection only when it provides reliable evidence for a better, reproducible, and generalizable decision, not merely when it correlates with observed MTL gain.

Their argument combines four separate steps: score, intervention effect, preferred configuration, routing gain. They test each step across several task pairs from different benchmark datasets (CelebA, QM9, 16 regression families) using a controlled set of configurations (separate encoders, frozen/pretrained/adapted shared encoders, residual output coupling). The findings are mixed.

Critical strengths
  • The paper is looking at an interesting gap between predicting a task relationship and using that prediction to make a useful decision.

  • Well-written paper with strong problem formulation and evaluation methodology.

    • Careful decomposition of affinity evaluation into distinct inferential steps.
    • Rather than stopping at correlation between an affinity score and measured MTL gains, the paper examines mechanism specificity, measurement reliability, configuration selection, and out-of-sample routing performance.
    • The residual-headroom check against a small portfolio (Eq. 6) is a practical, reusable baseline.
  • The framework's pre-specified primary hypotheses, reliability checks before interpreting nulls, variance decomposition over partitions vs. initializations, and candid reporting of where results fail are commendable, and they strengthen the evaluation.

Critical weaknesses
  • The practical impact of the paper is somewhat unclear. It is stronger at showing that existing validation practices are insufficient than at establishing when affinity-based selection gives a real advantage. The takeaway for practitioners (check residual headroom beyond a small fixed portfolio first) is useful, but it is mostly a reason for caution rather than a clear guideline.

  • The alternatives are not directly compared. The related work acknowledges that prior methods already go beyond score-gain correlation by predicting gains or constructing and evaluating groupings (Fifty et al., Song et al., Ayman et al.). The paper positions itself as complementary to these but does not evaluate them. The only group-level evidence is a simplified three-task CelebA experiment. The strongest claims therefore apply to the pairwise regime, which more recent methods were partly designed to move beyond.

  • A clearer comparison with these alternatives, and a more explicit discussion of the practical implications of the findings, would strengthen the paper's contribution.

Questions for authors

See Weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6uDV (1/2)

Thank you for recognizing the value of our evaluation framework and for the helpful suggestions on comparisons with existing methods and practical implications. We agree that the paper should explain more clearly how its findings relate to existing affinity-based approaches and how they can guide practical configuration decisions. We have added a TAG experiment, expanded the three-task analysis, and included a practical evaluation protocol.

Comparison with existing affinity methods (W2/W3) We added an evaluation of TAG's lookahead affinity score across all three domains (Appendix C.6, Tables 15–16). We compute the score using an SGD lookahead from a jointly trained model for each pair. This tests TAG's diagnostic, not the full task-grouping algorithm. The table below reports the SGD-lookahead results. We also test a simple decision rule of our own: use joint training when TAG affinity is positive and separate training otherwise.

DomainTAG–joint-training ρ\rhoPairs assigned joint trainingRule gain over fixed [95% CI]
CelebA+0.398*99.6%−0.0001 [−0.0003, +0.0000]
QM9+0.31891.1%−0.0708 [−0.1489, +0.0002]
Regression+0.05999.7%−0.0058 [−0.0293, +0.0124]

An asterisk indicates that the vertex-bootstrap 95% interval for the correlation excludes zero. The rule is compared against the leave-one-pair-out best fixed choice between separate and joint training.

On CelebA, TAG correlates with the observed benefit of joint training (ρ=+0.398\rho=+0.398), but the sign-based rule chooses joint training for almost every pair and does not improve on the best fixed choice. TAG is also associated with reuse (+0.275+0.275) and output coupling (+0.432+0.432), so it is not specific to joint training in this experiment.

We further tested whether TAG provides useful information for learned routing. Adding TAG as an additional feature improves CelebA held-out-task routing gain by +0.0017 [95% CI: +0.0007, +0.0027] with six independent-output configurations and +0.0023 [+0.0015, +0.0033] with all twelve. Some TAG variants provide positive incremental gains elsewhere, but they do not establish positive net routing gains over the fixed baseline outside CelebA. Since our TAG implementation requires a joint training run for each pair, these results show that the score contains additional useful information, but do not establish a computational advantage.

Together, these experiments strengthen our main conclusion: an affinity score may contain useful information about task interactions without directly providing a better configuration decision. They also show that such information can still be useful as part of a learned selector. We have also clarified the relationship with MTG-Net and ETAP in Sections 2 and 6, with full routing results in Appendix F.2 (Tables 29–30). These methods primarily address which tasks should be grouped, whereas our study focuses on how a given target–auxiliary pair should share representations and outputs. Our related routing rules test similar ideas but are not reproductions of those grouping algorithms.

Extending the analysis beyond task pairs (W2) Our original submission examined 100 CelebA triplets, each containing one target and two auxiliaries. We have extended this analysis with a group-level portfolio comparison (Appendix G, Tables 32–33).

Even when pairwise decisions use measured losses, they select the best group configuration on only 40% of triplets, compared with 38% for the best fixed group configuration. One reason is that the group oracle often prefers an encoder pretrained on both auxiliaries, a configuration that cannot be identified by selecting between individual pairwise alternatives. We then compared pairwise composition with a small group-level portfolio. A portfolio of three configurations leaves only 0.0008 residual oracle headroom, below the replicate noise level of 0.0084. More importantly, when portfolio selection and evaluation use independent data partitions, the portfolio achieves +0.0101 [95% CI: +0.0056, +0.0149] lower held-out NLL than pairwise composition.

These results suggest that pairwise decisions do not necessarily combine well when multiple auxiliaries are available, and that a small group-level portfolio may provide a useful alternative. However, the experiment is limited to CelebA triplets, so we do not claim that this resolves the general task-grouping problem.

AuthorsOpenReview ↗

Response to Reviewer 6uDV (2/2)

Practical implications (W1/W3) We agree that the paper should provide more than reasons to be cautious about affinity scores. Following your suggestion, we added Box 1 at the end of the main paper to summarize a practical procedure for evaluating affinity-guided configuration selection.

The protocol recommends estimating uncertainty across data partitions and initializations, checking how much oracle headroom remains beyond a small fixed portfolio, and testing whether a diagnostic reliably tracks its intended intervention effect. Because portfolio headroom uses an oracle member choice, independent validation is still needed to select a configuration in practice. Any learned selector should be evaluated on held-out tasks or families using the intended metric and accounting for its training cost.

These recommendations follow directly from our results. We see the practical contribution of our work as helping practitioners decide when an affinity score provides enough evidence to guide configuration selection, when a simpler portfolio may be sufficient, and when further validation is needed. The protocol does not guarantee that a learned selector will improve performance, but it helps assess whether one is likely to offer benefits over simpler alternatives.

Reviewer_6uDVOpenReview ↗

Response by Reviewer 6uDV

Thanks to the authors for the detailed responses and updated manuscript. I would like to keep my original positive score.

Reviewer_gMyaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether scores that estimate relationships between tasks can guide the choice of tasks to combine in multitask learning. For each pair of target and auxiliary tasks, this paper evaluates training under four types of models, including separate task-specific encoders, a shared encoder frozen at random initialization, a shared encoder pretrained on the auxiliary task and then frozen, and a shared encoder trained jointly on both tasks. Additionally, this paper adds task-specific adapters to the frozen pretrained encoder and fine-tuning of the auxiliary-pretrained encoder on the target. Then, under each setting, this paper tests predictions with and without the auxiliary task label at test time, for twelve configurations in total. Across 1,914 task pairs from CelebA, QM9, and multi-output regression datasets, some scores predict specific transfers, but those correlations do not consistently identify which of the twelve setups (including encoders and prediction settings) should be selected based on held-out target loss. Further, this paper proposes a selector method, which improves negative log-likelihood on CelebA tasks by 2.1% over the best fixed choice when only target inputs are used at test time. The selector does not establish a gain on held-out regression tasks.

Critical strengths
  • This paper finds that a score tracking one benefit need not identify the best training strategy. The comparisons are relatively extensive. On CelebA, the feature-transfer score remains correlated with frozen-feature gains across two backbones, while the ones for fine-tuning do not.
  • The evaluation includes generalization to held-out tasks, which considers gains on familiar task pairs and gains on new tasks.
Critical weaknesses
  • The algorithmic implications for multitask learning remain weak and mostly consider the case of combining two tasks. Most experiments choose a strategy for one target and one auxiliary task. In experiments with three tasks, pairwise choices match the best group configuration in 40% of cases, and their gain over that choice is not statistically significant. The implication of this analysis for better solving which tasks to train together is unclear.
  • Training the selector requires outcomes over 1,122 task pairs and about 5,715 model fits in the reported results. This paper does not yet establish a general method for task selection.
  • The writing is hard to follow. The abstract and introduction use undefined terms and lack explanation, for example, “configuration,” “intervention effect,” and “routing gain.”
Questions for authors
  1. What would be the practical implication for selecting groups of three or more tasks?
  2. From the existing results, how does that choice depend on the evaluation metric and the number of future tasks in selector training?
4 comments
AuthorsOpenReview ↗

Response to Reviewer gMya (1/2)

Thank you for the valuable suggestions on the practical implications of our work, especially its extension to larger task groups. We agree that understanding pairwise configuration choices does not directly solve more general task-grouping problems. Our goal is to examine what evidence is needed before using task-affinity scores to make such decisions, not to propose a general task-grouping algorithm. We have added experiments to better understand these limitations and the practical value of configuration selection.

What do the pairwise results imply for three or more tasks? (W1/Q1) We agree that pairwise configuration selection does not directly solve the problem of choosing larger task groups. Our original three-task experiment (Appendix G) showed this limitation: even when the pairwise rule uses measured losses instead of estimated affinity scores, it selects the best group configuration on only 40% of the 100 CelebA triplets, compared with 38% for the best fixed configuration. Its improvement over the fixed choice is not statistically resolved. One reason is that benefits from individual auxiliaries do not necessarily combine. Among triplets where both auxiliaries are individually beneficial, sharing with both outperforms the better single-auxiliary configuration on only 65%. Moreover, the group oracle selects an encoder pretrained on both auxiliaries on 38% of triplets, a configuration that has no direct pairwise counterpart.

To further address the practical implications, we added a group-level portfolio analysis (Appendix G, Table 33). A portfolio of three configurations reduces the remaining oracle headroom to 0.0008, below the replicate-scale uncertainty of 0.0084. Since this oracle comparison assumes that the best portfolio member is known, we also evaluated selection using independent data partitions. We select the portfolio and its configuration using two partitions and evaluate the selected model on a third, independently retrained partition. Under this protocol, the portfolio achieves lower held-out NLL than pairwise composition by +0.0101 [95% CI: +0.0056, +0.0149], performing better on 62% of triplets.

These results suggest a practical direction: when multiple auxiliaries are available, selecting them independently based on pairwise benefits may be insufficient. Evaluating a small portfolio of group-level configurations can provide a better alternative to composing pairwise decisions, although it requires additional group-level model evaluations. However, our experiments cover only CelebA triplets and six group configurations, so we present this as an initial extension of our pairwise analysis, not a general task-grouping algorithm.

Is training a selector worth the computational cost? (W2) We agree that the computational cost is an important consideration. Our cost analysis in Appendix F.4 estimates that collecting the selector's training outcomes requires approximately 5,715 model fits. For a new CelebA target with 34 possible auxiliaries, exhaustively evaluating the six input-only configurations requires roughly 170 fits, whereas the learned router requires only two additional target-side fits if the auxiliary-pretrained encoders are cached. Under these assumptions, the router saves approximately 168 fits per target and would recover its initial training cost after about 35 future targets.

This estimate is based on model-fit counts rather than measured wall-clock time, and depends on the reuse of pretrained encoders and the number of future targets. For a small number of targets, training a selector may not be worthwhile. We do not claim that our approach provides a generally efficient task-selection algorithm, but rather identify conditions under which learned routing may justify its cost.

AuthorsOpenReview ↗

Response to Reviewer gMya (2/2)

How do the evaluation metric and the amount of selector training data affect the results? (Q2) Our existing results showed that routing performance depends on the evaluation metric. On CelebA, the task-extrapolation routing gain is +0.0087 [95% CI: +0.0029, +0.0141] under NLL, but only +0.0003 [−0.0020, +0.0023] under 1−AUC1-\mathrm{AUC}, with a confidence interval that includes zero (Appendix F.4). The best fixed configuration also changes with the metric. Thus, an improvement under NLL does not necessarily translate into better performance under ROC-AUC.

To further address the reviewer's question, we added a task-level learning-curve experiment examining how routing performance depends on the number of previously evaluated tasks available for training the selector (Appendix F.6, Table 31). We vary the number of training tasks while keeping the held-out evaluation tasks and comparison baseline unchanged.

Previously evaluated tasks mmA6ind\mathcal A_6^{\mathrm{ind}}: gain [95% CI]A12\mathcal A_{12}: gain [95% CI]
8+0.0054 [−0.0017, +0.0121]+0.0051 [−0.0023, +0.0133]
12+0.0071 [+0.0004, +0.0135]+0.0073 [−0.0007, +0.0159]
20+0.0085 [+0.0023, +0.0146]+0.0087 [+0.0002, +0.0179]
24+0.0091 [+0.0033, +0.0146]+0.0094 [+0.0004, +0.0192]

Selected CelebA rows from Table 31. Bold indicates a 95% confidence interval excluding zero; gains are evaluated on held-out tasks.

Routing gains generally increase as more previously evaluated tasks become available. Under the six input-only configurations, the confidence interval first excludes zero at m=12m=12; under all twelve configurations, this occurs at m=20m=20. These values are specific to CelebA, not universal training-data requirements.

Together, these analyses show that the practical value of learned routing depends on the evaluation metric, the amount of training data available for the selector, and whether it will be applied to enough future targets to justify its computational cost.

Writing improvement (W3) We have revised the abstract, introduction, and setup to explain the main concepts earlier, added a schematic overview of the configurations (Figure 1), and reorganized the appendices to better follow the main paper. We also included a practical evaluation protocol (Box 1), which summarizes what to check before using affinity scores to guide configuration decisions.

Thanks to the authors for the detailed responses. However, the responses have not addressed my question: Is there a principled method in this paper's analyses for conducting multitask learning (or selecting tasks to combine in the joint training setup)? The algorithmic implication of this paper is unclear. Thus, I would like to maintain my original evaluation.

Thank you for your quick response. We would like to clarify that our paper does not propose a general task-grouping algorithm. Instead, we study a complementary question: given a target and a candidate auxiliary task, when is there sufficient evidence to choose a particular multi-task configuration?

We have better clarified this distinction in the revised manuscript and added a practical evaluation procedure (Box 1) at the end of the main paper. For your convenience, we summarize the procedure as follows:

  1. Measure experimental uncertainty across data partitions and random seeds.
  2. Assess whether selection is worthwhile by estimating the remaining performance headroom beyond a small fixed portfolio of configurations, taking the estimated uncertainty into account.
  3. Determine whether affinity provides useful evidence by testing whether a diagnostic reliably predicts the relevant intervention effect, rather than assuming that affinity directly implies a better configuration.
  4. Validate the resulting decisions by evaluating a learned configuration selector against strong fixed alternatives on held-out tasks or families, considering the evaluation metric and computational cost. If meaningful selection gains cannot be established, a simpler fixed configuration or independently validated portfolio may be sufficient.

Our contribution is not a new task-grouping algorithm, but a principled framework for evaluating when affinity-guided configuration selection is justified and when simpler alternatives may be preferable. We hope this clarification helps explain the practical implications of our work and addresses your concern.

Reviewer_asg8OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper provides an in-depth study of task affinity measures, examining their relationship with performance gains and their usefulness for selecting optimal configurations (what and how they should share) and how these “optimal” choices survive across other contexts. The motivation is well justified: rather than proposing another measure or algorithm, the paper seeks to clarify what existing scores actually capture and which decisions they can support. In particular, it distinguishes between a score’s association with the effect of an intervention, its usefulness for selecting an action, and the extent to which these relationships generalize to other task pairs or experimental settings, while accounting for potential confounders such as the chosen backbone and evaluation loss.

Critical strengths
  • 1. The paper addresses an important question by systematically analysing existing task affinity measures in MTL, rather than simply proposing a new method and demonstrating its effectiveness. I found this focus on understanding what existing measures capture and when they are useful particularly interesting and necessary.

  • 2. The paper analyses a broad range of interventions across many task pairs and examines numerous potential confounders that could influence the measured relationships. The results appear to be reported fairly and transparently, including when the findings are inconclusive.

Critical weaknesses
  • 1. The paper is not yet sufficiently clear. The introduction refers to several equations well before they are introduced, and the specific configurations, such as separate encoders, shared encoders, and encoders frozen at initialisation, are not sufficiently explained, making the paper difficult to follow and interpret. It would help to distinguish more clearly between simultaneous learning, transfer learning, and coupled learning, and to cite work on the specific configurations listed in Table 1, particularly since “multi-task learning” is sometimes used to refer only to simultaneous learning. A schematic overview of these configurations, along with a clearer explanation of how results are averaged across task pairs, would also help the reader.

  • 2. Currently, the relationships between affinity scores and intervention effects are treated as independent of training time, although task interactions are generally understood to evolve during training.

Questions for authors
  • Do you expect the results to change if the correlations are evaluated at different points during training? Could some scores be more predictive of intervention effects early in training? How were the training duration and number of training samples chosen?

  • The differences across datasets suggest that the effects of interventions may also depend on the data. For example, Table 2 shows that frozen transfer is never selected for QM9 but is selected for approximately 20% of the regression pairs. Do you have a plausible explanation for this difference? Theoretical work on continual, transfer, and multi-task learning suggests that task similarity plays a role. Given this potential dependence, is averaging across task pairs justified?

  • How do you measure complexity? (equation 3)

2 comments
AuthorsOpenReview ↗

Response to Reviewer asg8 (1/2)

Thank you for recognizing the value of our work and for the helpful suggestions on improving clarity and examining the training time effect. We have revised the presentation and added a training-checkpoint experiment to address these concerns. We respond to your comments below.

Improved paper representation and clarity (W1) We have introduced the main concepts earlier in the abstract and introduction and clarified their definitions in Section 3. We also replaced the original configuration table with a schematic overview (Figure 1), showing six encoder strategies with either independent or coupled outputs. The figure distinguishes simultaneous joint training from frozen representation reuse and adaptation after transfer, and illustrates the corresponding intervention effects. We also clarify that coupled configurations require the auxiliary outcome at inference, whereas independent-output configurations predict from the input alone. We have further clarified how these configurations relate to existing work in Section 2. Together with the experimental details in Appendix A, these changes should make the different configurations and their comparisons easier to follow.

Do task relationships change during training? (W2/Q1) Yes. We added a CelebA checkpoint experiment evaluating the reuse probe, intervention effects, and preferred configurations at 2, 5, 10, and 20 epochs (Appendix E.7, Table 27). Auxiliary pretraining and downstream training use matched checkpoint budgets, while calibration and the near-tie tolerance are recomputed at each checkpoint.

EpochsMean reuse effectMean adaptation-after-transfer effectProbe–reuse ρ\rhoFrozen transfer preferred
2+0.005+0.000+0.046%
5+0.014−0.001+0.30*19%
10+0.016−0.007+0.40*23%
20−0.010+0.013+0.56*3%

The adaptation column reports the controlled effect of adaptation after transfer. Correlations compare the probe and reuse effect at the same checkpoint. An asterisk indicates that the vertex-bootstrap 95% confidence interval excludes zero. Preference rates use the six checkpointed configurations, not the full twelve-configuration space.

The probe–reuse association becomes stronger during training, increasing from +0.04 to +0.56. However, the average reuse effect changes sign by epoch 20, and frozen-transfer selection drops from 23% at epoch 10 to 3% at epoch 20. We also checked whether earlier probe values are informative about the final reuse effect. The correlations are +0.19, +0.33, and +0.48 at epochs 2, 5, and 10, respectively, compared with +0.56 at epoch 20. These results show that useful information can emerge before training finishes, but the intervention benefits and preferred configurations can change with training time. They further support our main conclusion that an informative affinity score does not necessarily determine the preferred configuration. The experiment is limited to CelebA and one optimizer and learning-rate schedule, so we do not assume the same pattern holds across all settings.

How were training budgets and data sizes chosen? (Q1) We used fixed training budgets of 20 epochs for CelebA, 40 for QM9, and 300 for regression, instead of tuning early stopping separately for each configuration (Appendix A.2). The data sizes were chosen to study transfer in data-limited settings: 200 target and 4,000 auxiliary examples for CelebA, 8,000 molecules for QM9 fitting, and 60 rows per regression target. We do not claim these settings are optimal. Our original experiments examined the influence of target-data availability (Appendix E.6, Table 26) and update-matched joint training (Appendix E.4). These analyses showed that configuration preferences can change with target-data availability and competitor optimization, even when the association between a diagnostic and its intended intervention effect remains similar. The new checkpoint experiment (Appendix E.7) extends this analysis by showing that training duration also affects both intervention benefits and configuration preferences.

AuthorsOpenReview ↗

Response to Reviewer asg8 (2/2)

Dataset differences and aggregation across task pairs (Q2) We agree that intervention effects can vary across datasets and task pairs, and that task similarity may contribute to these differences. However, the preferred configuration also depends on the size and uncertainty of the performance differences. For example, on QM9, frozen transfer has a median loss gap of about 0.94 nats from the best configuration and is never preferred. On the regression families, it achieves the exact minimum loss on only 3.5% of pairs, but is preferred on 6–9% under the noise-aware selection rule because more configurations fall within the near-tie tolerance. The predefined parsimony order then determines which configuration is selected. We also distinguish frozen transfer from adapted transfer, which is preferred on 23.7% of regression pairs under the expanded configuration set (Section 5 and Appendix E.3). These results help explain the different selection rates, although we cannot attribute the differences to task similarity alone. They show that configuration preferences depend on both the task relationship and the training and evaluation context.

Regarding aggregation, we report results separately for CelebA, QM9, and the regression families rather than pooling pairs across domains. For score–effect associations, we rank auxiliaries within each target, center the ranks, and then pool them across targets. This evaluates whether a diagnostic ranks auxiliaries for the same target, rather than simply distinguishing targets with different overall performance. We also report mean intervention effects together with the fraction of pairs showing positive effects, so the conclusions do not rely on averages alone. We use task-level vertex resampling to account for dependencies between pairs sharing a task and family-level resampling for cross-family inference. Appendix B further examines uncertainty across data partitions and initializations. Thus, our aggregated results summarize patterns within each domain without assuming that intervention effects are uniform across task pairs or dataset families.

How is complexity measured? (Q3) We agree that the term "complexity" was not very clear. We do not measure complexity using FLOPs, parameter count, or training time. Instead, we use a predefined ordinal parsimony order to choose among configurations whose observed losses fall within the replicate-scale tolerance of the minimum. We have clarified this definition in Eq. (3) and Appendix E.3. Our sensitivity analysis also considers alternative orders, including selecting the exact minimum-loss configuration (Table 22). Frozen-transfer selection changes by at most two percentage points on CelebA and three on regression, and remains zero on QM9. This shows that the main findings are not driven by the particular tie-breaking order.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for their thoughtful feedback. We agree that the paper needed a clearer explanation of the configurations, stronger evidence about practical decisions, and a better discussion of the limits of pairwise analysis. In the revised manuscript, we have improved the presentation and expanded several analyses to address these concerns.

To make the paper easier to follow, we have:

  • Added an overview of the configurations (Figure 1), showing how they differ in representation sharing and output coupling, together with corresponding intervention effects.
  • Added a practical evaluation protocol (Box 1) explaining what to check before using an affinity score to select configurations.
  • Explained the main concepts earlier in the abstract and introduction.
  • Better clarified how results are aggregated across task pairs, how uncertainty is estimated, and how preferred configurations are selected when performance differences are small.
  • Reorganized some appendices to better follow the main paper and make the supporting analyses easier to find.

To address the reviewers' concerns, we have added the following analyses:

  • Training-budget analysis (Reviewer asg8): We measure the reuse probe and intervention effects at different training checkpoints. The probe becomes more informative during training, but the intervention benefits and preferred configurations also change. This strengthens our conclusion that a diagnostic can remain informative while the configuration decision depends on the training setting (Appendix E.7, Table 27).
  • Selector training-data analysis (Reviewer gMya): We vary the number of previously evaluated tasks used to train the router. More training tasks generally improve routing gains on CelebA, showing how the amount of available training data affects the usefulness of learned selection. We also clarify the evaluation-metric analysis already present in the original paper: routing gains are statistically resolved under NLL but not under 1−AUC1-\mathrm{AUC}. Together, these results show that routing value depends on both the available training data and the evaluation metric (Appendices F.4 and F.6, Table 31).
  • Larger task groups (Reviewers gMya and 6uDV): We extend the original three-task analysis by evaluating a small group-level portfolio. The original results showed that pairwise decisions do not always combine well; the new experiments show that the portfolio achieves better held-out performance under independent evaluation. This provides a practical alternative worth investigating when multiple auxiliaries are available, while confirming that our pairwise findings do not automatically extend to larger groups (Appendix G, Tables 32–33).
  • Comparison with existing affinity methods (Reviewer 6uDV): We evaluate TAG's lookahead score as a diagnostic and an additional routing feature, and clarify how our routing approaches relate to MTG-Net and ETAP. TAG provides useful information about joint-training benefits on CelebA, but a simple rule based on its sign does not outperform the best fixed configuration. Adding TAG to the learned router gives a small additional improvement on CelebA. These results further support our main argument that an affinity score can be informative without being sufficient on its own to make better configuration decisions (Appendix C.6, Tables 15–16).

Overall, the additional experiments strengthen our main conclusions. They show more clearly that the usefulness of an affinity score depends not only on what it measures, but also on the training conditions, available configurations, evaluation metric, and generalization setting. They also clarify the limits of our findings, particularly when moving from pairs to larger task groups or from previously evaluated tasks to new ones.

Correction to the routing evaluation. During our experimental checks, we found that the abstention threshold had been selected using held-out folds and corrected the procedure. After rerunning the experiments, the main CelebA task-extrapolation gain changed slightly from +0.0099 to +0.0087, without changing our conclusion. We also clarified the best fixed baseline and added a deployment-matched comparison (Appendix F.2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.