- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Task-affinity scores are normally validated by correlating them with observed multi-task gain. The authors argue that task affinity justifies pairwise MTL configuration selection only when it provides reliable evidence for a better, reproducible, and generalizable decision, not merely when it correlates with observed MTL gain.
Their argument combines four separate steps: score, intervention effect, preferred configuration, routing gain. They test each step across several task pairs from different benchmark datasets (CelebA, QM9, 16 regression families) using a controlled set of configurations (separate encoders, frozen/pretrained/adapted shared encoders, residual output coupling). The findings are mixed.
Critical strengths
-
The paper is looking at an interesting gap between predicting a task relationship and using that prediction to make a useful decision.
-
Well-written paper with strong problem formulation and evaluation methodology.
- Careful decomposition of affinity evaluation into distinct inferential steps.
- Rather than stopping at correlation between an affinity score and measured MTL gains, the paper examines mechanism specificity, measurement reliability, configuration selection, and out-of-sample routing performance.
- The residual-headroom check against a small portfolio (Eq. 6) is a practical, reusable baseline.
-
The framework's pre-specified primary hypotheses, reliability checks before interpreting nulls, variance decomposition over partitions vs. initializations, and candid reporting of where results fail are commendable, and they strengthen the evaluation.
Critical weaknesses
-
The practical impact of the paper is somewhat unclear. It is stronger at showing that existing validation practices are insufficient than at establishing when affinity-based selection gives a real advantage. The takeaway for practitioners (check residual headroom beyond a small fixed portfolio first) is useful, but it is mostly a reason for caution rather than a clear guideline.
-
The alternatives are not directly compared. The related work acknowledges that prior methods already go beyond score-gain correlation by predicting gains or constructing and evaluating groupings (Fifty et al., Song et al., Ayman et al.). The paper positions itself as complementary to these but does not evaluate them. The only group-level evidence is a simplified three-task CelebA experiment. The strongest claims therefore apply to the pairwise regime, which more recent methods were partly designed to move beyond.
-
A clearer comparison with these alternatives, and a more explicit discussion of the practical implications of the findings, would strengthen the paper's contribution.
Questions for authors
See Weakness
3 comments
Response to Reviewer 6uDV (2/2)
Practical implications (W1/W3) We agree that the paper should provide more than reasons to be cautious about affinity scores. Following your suggestion, we added Box 1 at the end of the main paper to summarize a practical procedure for evaluating affinity-guided configuration selection.
The protocol recommends estimating uncertainty across data partitions and initializations, checking how much oracle headroom remains beyond a small fixed portfolio, and testing whether a diagnostic reliably tracks its intended intervention effect. Because portfolio headroom uses an oracle member choice, independent validation is still needed to select a configuration in practice. Any learned selector should be evaluated on held-out tasks or families using the intended metric and accounting for its training cost.
These recommendations follow directly from our results. We see the practical contribution of our work as helping practitioners decide when an affinity score provides enough evidence to guide configuration selection, when a simpler portfolio may be sufficient, and when further validation is needed. The protocol does not guarantee that a learned selector will improve performance, but it helps assess whether one is likely to offer benefits over simpler alternatives.
Response by Reviewer 6uDV
Thanks to the authors for the detailed responses and updated manuscript. I would like to keep my original positive score.
Response to Reviewer 6uDV (1/2)
Thank you for recognizing the value of our evaluation framework and for the helpful suggestions on comparisons with existing methods and practical implications. We agree that the paper should explain more clearly how its findings relate to existing affinity-based approaches and how they can guide practical configuration decisions. We have added a TAG experiment, expanded the three-task analysis, and included a practical evaluation protocol.
Comparison with existing affinity methods (W2/W3) We added an evaluation of TAG's lookahead affinity score across all three domains (Appendix C.6, Tables 15–16). We compute the score using an SGD lookahead from a jointly trained model for each pair. This tests TAG's diagnostic, not the full task-grouping algorithm. The table below reports the SGD-lookahead results. We also test a simple decision rule of our own: use joint training when TAG affinity is positive and separate training otherwise.
An asterisk indicates that the vertex-bootstrap 95% interval for the correlation excludes zero. The rule is compared against the leave-one-pair-out best fixed choice between separate and joint training.
On CelebA, TAG correlates with the observed benefit of joint training (ρ=+0.398), but the sign-based rule chooses joint training for almost every pair and does not improve on the best fixed choice. TAG is also associated with reuse (+0.275) and output coupling (+0.432), so it is not specific to joint training in this experiment.
We further tested whether TAG provides useful information for learned routing. Adding TAG as an additional feature improves CelebA held-out-task routing gain by +0.0017 [95% CI: +0.0007, +0.0027] with six independent-output configurations and +0.0023 [+0.0015, +0.0033] with all twelve. Some TAG variants provide positive incremental gains elsewhere, but they do not establish positive net routing gains over the fixed baseline outside CelebA. Since our TAG implementation requires a joint training run for each pair, these results show that the score contains additional useful information, but do not establish a computational advantage.
Together, these experiments strengthen our main conclusion: an affinity score may contain useful information about task interactions without directly providing a better configuration decision. They also show that such information can still be useful as part of a learned selector. We have also clarified the relationship with MTG-Net and ETAP in Sections 2 and 6, with full routing results in Appendix F.2 (Tables 29–30). These methods primarily address which tasks should be grouped, whereas our study focuses on how a given target–auxiliary pair should share representations and outputs. Our related routing rules test similar ideas but are not reproductions of those grouping algorithms.
Extending the analysis beyond task pairs (W2) Our original submission examined 100 CelebA triplets, each containing one target and two auxiliaries. We have extended this analysis with a group-level portfolio comparison (Appendix G, Tables 32–33).
Even when pairwise decisions use measured losses, they select the best group configuration on only 40% of triplets, compared with 38% for the best fixed group configuration. One reason is that the group oracle often prefers an encoder pretrained on both auxiliaries, a configuration that cannot be identified by selecting between individual pairwise alternatives. We then compared pairwise composition with a small group-level portfolio. A portfolio of three configurations leaves only 0.0008 residual oracle headroom, below the replicate noise level of 0.0084. More importantly, when portfolio selection and evaluation use independent data partitions, the portfolio achieves +0.0101 [95% CI: +0.0056, +0.0149] lower held-out NLL than pairwise composition.
These results suggest that pairwise decisions do not necessarily combine well when multiple auxiliaries are available, and that a small group-level portfolio may provide a useful alternative. However, the experiment is limited to CelebA triplets, so we do not claim that this resolves the general task-grouping problem.