- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes to optimize knowledge sharing in multi-task learning as a resource allocation problem, and solve it using optimal transport. The results show that the method works well on medical domain as well as a standard CelebA task.
Critical strengths
The approach is compelling and clear. The improvements are rather small but consistent with what can be assumed in MTL.
Critical weaknesses
The fact that transport works and results in differences in modules is an intriguing result, but it is really a starting point. What did they learn that's different from module to module? Why do certain tasks rely on some modules and not others? The analysis and ablation study are good, but they do not improve our understanding of what makes MTL work.
The list of contributions is a summary of what you did---it does not specify what the study actually contreibutes to science. For instance, introducing a method is not a contribution. Even if that method improves performance, it is not quite a contribution. If you identify a problem with the current state of the art, design a method to solve that problem, and demonstrate that it does, then it is a contribution: it results in an understanding that wasn't there before. That's how you need to phrase the contributions.
Questions for authors
The main question is described above: how and why does it work?
Also, there's some multitask work that suggests that learning across very different domains can help significantly (e.g. Meyerson NeurIPS 2019). Here the tasks are rather similar. Would the method work across e.g. langauge, vision, DNA analysis tasks? Why or why not?
4 comments
Part 2/2
Taken together, these findings suggest that ASK-OT works through three complementary mechanisms: constrained module capacity encourages specialization; task-dependent allocation enables asymmetric sharing of these specialized modules; and input-dependent adjustments adapt this sharing to variation within each task.
Table: KM usage per task on the FA test set. For each task, the top three KMs are sorted by their mean transport-plan weight averaged over images. The percentage in brackets is the share of images for which that KM receives the largest weight.
| Task | #1 | #2 | #3 |
|---|---|---|---|
| CNV | KM3 0.304 (62%) | KM4 0.203 (35%) | KM1 0.056 (3%) |
| HyperF Area | KM2 1.042 (76%) | KM4 0.248 (24%) | KM3 0.018 (0%) |
| HyperF Type | KM4 0.769 (69%) | KM3 0.350 (23%) | KM2 0.111 (1%) |
| HyperF Fovea | KM4 0.277 (49%) | KM3 0.205 (48%) | KM2 0.064 (3%) |
| HypoF Type | KM5 0.495 (86%) | KM3 0.192 (13%) | KM2 0.086 (1%) |
| HypoF Area | KM5 0.539 (74%) | KM3 0.177 (3%) | KM2 0.149 (22%) |
| HypoF Fovea | KM5 0.376 (58%) | KM3 0.304 (41%) | KM2 0.093 (0%) |
| DR | KM1 1.455 (100%) | KM3 0.050 (0%) | KM5 0.021 (0%) |
We added these analyses to the revised paper (sec.6.2, Task-level and input-dependent allocation)
Q2 (Learning across very different domains): We thank the reviewer for pointing us to this work. Meyerson & Miikkulainen (2019) study knowledge sharing across tasks with different inputs and architectures (vision, language, and genomics). ASK-OT addresses a different setting: multiple tasks defined on the same input, as is common in medical imaging, where several clinical findings are assessed from the same study. Importantly, sharing an input does not imply that tasks benefit equally from shared representations. As shown in Section 3, even clinically related FA tasks exhibit asymmetric and sometimes negative transfer. Extending ASK-OT to more diverse domains would depend on how the data are structured:
Paired modalities. When multiple modalities are available for the same case (e.g., fundus images and OCT scans from the same eye), the framework could be extended using modality-specific encoders and shared knowledge modules. The transport plan could then learn how different tasks allocate these modules, potentially capturing asymmetric sharing across modalities.
Unpaired domains. For tasks such as language, vision, and DNA analysis, there may be no common input on which all tasks are defined. In this setting, modality-specific encoders would require a shared representation space for their knowledge modules to be transferable across domains. This would also require adapting the allocation mechanism: unlike our current setting, an input from one domain cannot jointly inform the allocation across tasks from other domains. Thus, while the OT-based resource allocation principle is not inherently tied to a particular domain, extending the current input-dependent sharing mechanism to unpaired domains would require additional design choices, particularly for cross-domain representation alignment and allocation.
Thanks for the answer, but I'd still like to see a high-level insight on what's been learned, in addition to the technical details. I still have just bits and pieces, no the big picture. What problem in knowledge sharing does the reseach allocation solve?
High-Level Insight
We thank the reviewer for the follow-up. In one sentence, resource allocation solves the problem of sharing knowledge between tasks whose relationships are asymmetric, without either forcing all tasks onto one representation or fixing in advance which tasks share with which. We summarize the big picture below; this will be added to the final manuscript.
Problem. When all tasks share one representation, their needs differ: some can benefit from overlapping information, potentially in asymmetric ways, while others need different information to be emphasized. A single representation used in the same way by all tasks cannot adequately accommodate these differences.
Our view. We build on a shared encoder that provides a common representation for all tasks, what differs between tasks is how this representation needs to be transformed and used. Knowledge modules (KMs) provide this flexibility: they are nonlinear transformations that recombine the shared representation into distinct views, each emphasizing different information (Fig. 2a). The question then becomes which tasks should use which views, and to what extent. We formulate this as a resource allocation problem: a limited set of knowledge modules must serve tasks with different representational needs.
What resource allocation solves. Simply offering multiple views is not enough, because left unconstrained they tend to become redundant. Allocating limited capacity across tasks forces the views to become distinct, and lets each task draw on them to its own degree. The sharing structure therefore emerges from the tasks’ actual needs rather than being imposed in advance.
What we learned. Shared knowledge is more effective when organized into specialized views that tasks can draw on to different degrees, rather than as a single representation used equally by all tasks. This allows knowledge sharing to reflect the actual, often asymmetric, relationships between tasks.
Part 1/2
We thank the reviewer for the constructive feedback and reviews. Below, we address questions, which have been incorporated into the revised paper.
Contribution list re-framing: We thank the reviewer for this feedback, which allowed us to revisit the paper’s narrative and elicit its fundamental contributions. Following the suggestion, we have rephrased the contributions to state the problem we identify, how we address it, and what we show. The revised list reads as follows:
Our contributions are threefold: (1) Relatedness does not imply symmetric transfer. We study a set of clinically related tasks and find that transfer between them is not predicted by clinical relatedness. It is often asymmetric and sometimes one-directional, and no fixed task grouping is optimal for all tasks. This challenges MTL approaches that share knowledge symmetrically or within fixed task groups. (2) Asymmetric sharing can be learned as an allocation problem. We show that casting representation sharing as optimal transport between knowledge modules and tasks lets the sharing structure emerge rather than be fixed a priori. MCC enforces that every module is shared across tasks, while TCR lets each task draw on modules to different degrees. We show that under these constraints the sharing between two tasks can be asymmetric. We instantiate this formulation as ASK-OT (ASymmetric Knowledge sharing via Optimal Transport). (3) Empirical validation and analysis. We provide theoretical insights into the asymmetric knowledge sharing of ASK-OT, and evaluate it experimentally across medical and general-domain benchmarks, showing that it outperforms representative baselines across multiple MTL paradigms.
Q1 (How and why does it work?): We thank the reviewer for raising this important question. Our analysis provides four complementary insights.
(1) Asymmetric transfer motivates allocation rather than grouping. Our transfer analysis (Section 3) shows that clinical relatedness alone does not predict transfer: transfer between two tasks can be unequal or even one-sided. Discrete task groups impose symmetric sharing and cannot capture these patterns. ASK-OT instead allows each task to draw differently from shared modules, accommodating asymmetric relationships between tasks.
(2) Modules develop different specializations. MCC constrains the total allocation to each module, while the transport costs determine how this capacity is distributed across tasks. As a result, modules receive different combinations of task gradients during training. Consistent with this specialization, module-output similarity decreases during ASK-OT training, whereas under uniform assignment the modules converge to nearly identical transformations (Figure 5a). The attention maps added in Section 6.2 provide further qualitative evidence: different modules emphasize different image regions. For example, on FA, KM3 and KM4 focus on hyperfluorescent lesions, whereas KM1 primarily attends to the peripheral retina and vasculature.
(3) Task-level allocation reflects these different specializations. The table below summarizes the most-used modules for each FA task. The three hypofluorescence tasks primarily rely on KM5, while HyperF Type and HyperF Fovea mainly use KM4 and KM3. DR relies almost exclusively on KM1. These patterns are consistent with our transfer analysis: CNV and HyperF Type draw on each other's preferred modules, in line with their mutual positive transfer, whereas the HypoF tasks draw on CNV's preferred module (KM3), but CNV draws little from their preferred module (KM5), consistent with the asymmetric CNV-HypoF transfer (observed in section3:Preliminaries).
(4) Input-dependent allocation adjusts this task-level structure. Since the transport cost depends on the module outputs, the plan is computed separately for each input. The transport plan average over all images captures the common task-level structure, while individual plans allow allocation to vary across inputs within the same task.
To illustrate this variation, we provide a qualitative case study of HyperF Type, whose five classes (None, Leakage, Pooling, Staining, Window Defect) represent different image patterns. For one test image per class, Section 6.2 (New paragraph: Task-level and input-dependent allocation) shows the per-input transport weights and module attention maps. Abnormal examples primarily use the modules favored by the task-level allocation (KM4 and KM3), but their relative contributions and attended regions vary with the lesion type. In contrast, normal image, which has no focal lesion, is handled differently: the allocation is spread over several modules, and the attention maps cover broad areas of the retina rather than a single region. We interpret this as a learned strategy: confirming the absence of hyperfluorescence requires inspecting the whole image, whereas detecting an abnormality only requires focusing on it.