acceptodds
Under review as a conference paper at ICLR 2027

Learning How to Share: Optimal Transport for Medical Multi-Task Learning

Abstract

We present ASK-OT (**AS**ymmetric **K**nowledge sharing via **O**ptimal **T**ransport), an innovative framework for multi-task learning that formulates knowledge sharing as a resource allocation problem. ASK-OT learns a set of knowledge modules from a shared representation and allocates them to tasks according to their representational needs. We formulate this allocation as an optimal transport problem, allowing task-specific and asymmetric sharing patterns to emerge from the learned representations rather than being imposed by predefined task groups. To account for the distinct roles of modules and tasks, we introduce a module-capacity constraint and task constraint relaxation, which respectively allow modules to be shared across tasks while allowing tasks to draw different amounts of knowledge. We theoretically characterize the resulting asymmetric sharing and demonstrate experimentally that ASK-OT outperforms representative baselines. Code will be available upon acceptance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multi-Task Learning, Optimal Transport, Representation Learning, Knowledge Sharing, Medical Imaging

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iUuuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes to optimize knowledge sharing in multi-task learning as a resource allocation problem, and solve it using optimal transport. The results show that the method works well on medical domain as well as a standard CelebA task.

Critical strengths

The approach is compelling and clear. The improvements are rather small but consistent with what can be assumed in MTL.

Critical weaknesses

The fact that transport works and results in differences in modules is an intriguing result, but it is really a starting point. What did they learn that's different from module to module? Why do certain tasks rely on some modules and not others? The analysis and ablation study are good, but they do not improve our understanding of what makes MTL work.

The list of contributions is a summary of what you did---it does not specify what the study actually contreibutes to science. For instance, introducing a method is not a contribution. Even if that method improves performance, it is not quite a contribution. If you identify a problem with the current state of the art, design a method to solve that problem, and demonstrate that it does, then it is a contribution: it results in an understanding that wasn't there before. That's how you need to phrase the contributions.

Questions for authors

The main question is described above: how and why does it work?

Also, there's some multitask work that suggests that learning across very different domains can help significantly (e.g. Meyerson NeurIPS 2019). Here the tasks are rather similar. Would the method work across e.g. langauge, vision, DNA analysis tasks? Why or why not?

4 comments

We thank the reviewer for the constructive feedback and reviews. Below, we address questions, which have been incorporated into the revised paper.

Contribution list re-framing: We thank the reviewer for this feedback, which allowed us to revisit the paper’s narrative and elicit its fundamental contributions. Following the suggestion, we have rephrased the contributions to state the problem we identify, how we address it, and what we show. The revised list reads as follows:

Our contributions are threefold: (1) Relatedness does not imply symmetric transfer. We study a set of clinically related tasks and find that transfer between them is not predicted by clinical relatedness. It is often asymmetric and sometimes one-directional, and no fixed task grouping is optimal for all tasks. This challenges MTL approaches that share knowledge symmetrically or within fixed task groups. (2) Asymmetric sharing can be learned as an allocation problem. We show that casting representation sharing as optimal transport between knowledge modules and tasks lets the sharing structure emerge rather than be fixed a priori. MCC enforces that every module is shared across tasks, while TCR lets each task draw on modules to different degrees. We show that under these constraints the sharing between two tasks can be asymmetric. We instantiate this formulation as ASK-OT (ASymmetric Knowledge sharing via Optimal Transport). (3) Empirical validation and analysis. We provide theoretical insights into the asymmetric knowledge sharing of ASK-OT, and evaluate it experimentally across medical and general-domain benchmarks, showing that it outperforms representative baselines across multiple MTL paradigms.

Q1 (How and why does it work?): We thank the reviewer for raising this important question. Our analysis provides four complementary insights.

(1) Asymmetric transfer motivates allocation rather than grouping. Our transfer analysis (Section 3) shows that clinical relatedness alone does not predict transfer: transfer between two tasks can be unequal or even one-sided. Discrete task groups impose symmetric sharing and cannot capture these patterns. ASK-OT instead allows each task to draw differently from shared modules, accommodating asymmetric relationships between tasks.

(2) Modules develop different specializations. MCC constrains the total allocation to each module, while the transport costs determine how this capacity is distributed across tasks. As a result, modules receive different combinations of task gradients during training. Consistent with this specialization, module-output similarity decreases during ASK-OT training, whereas under uniform assignment the modules converge to nearly identical transformations (Figure 5a). The attention maps added in Section 6.2 provide further qualitative evidence: different modules emphasize different image regions. For example, on FA, KM3 and KM4 focus on hyperfluorescent lesions, whereas KM1 primarily attends to the peripheral retina and vasculature.

(3) Task-level allocation reflects these different specializations. The table below summarizes the most-used modules for each FA task. The three hypofluorescence tasks primarily rely on KM5, while HyperF Type and HyperF Fovea mainly use KM4 and KM3. DR relies almost exclusively on KM1. These patterns are consistent with our transfer analysis: CNV and HyperF Type draw on each other's preferred modules, in line with their mutual positive transfer, whereas the HypoF tasks draw on CNV's preferred module (KM3), but CNV draws little from their preferred module (KM5), consistent with the asymmetric CNV-HypoF transfer (observed in section3:Preliminaries).

(4) Input-dependent allocation adjusts this task-level structure. Since the transport cost depends on the module outputs, the plan is computed separately for each input. The transport plan average over all images captures the common task-level structure, while individual plans allow allocation to vary across inputs within the same task.

To illustrate this variation, we provide a qualitative case study of HyperF Type, whose five classes (None, Leakage, Pooling, Staining, Window Defect) represent different image patterns. For one test image per class, Section 6.2 (New paragraph: Task-level and input-dependent allocation) shows the per-input transport weights and module attention maps. Abnormal examples primarily use the modules favored by the task-level allocation (KM4 and KM3), but their relative contributions and attended regions vary with the lesion type. In contrast, normal image, which has no focal lesion, is handled differently: the allocation is spread over several modules, and the attention maps cover broad areas of the retina rather than a single region. We interpret this as a learned strategy: confirming the absence of hyperfluorescence requires inspecting the whole image, whereas detecting an abnormality only requires focusing on it.

Taken together, these findings suggest that ASK-OT works through three complementary mechanisms: constrained module capacity encourages specialization; task-dependent allocation enables asymmetric sharing of these specialized modules; and input-dependent adjustments adapt this sharing to variation within each task.

Table: KM usage per task on the FA test set. For each task, the top three KMs are sorted by their mean transport-plan weight averaged over images. The percentage in brackets is the share of images for which that KM receives the largest weight.

Task#1#2#3
CNVKM3 0.304 (62%)KM4 0.203 (35%)KM1 0.056 (3%)
HyperF AreaKM2 1.042 (76%)KM4 0.248 (24%)KM3 0.018 (0%)
HyperF TypeKM4 0.769 (69%)KM3 0.350 (23%)KM2 0.111 (1%)
HyperF FoveaKM4 0.277 (49%)KM3 0.205 (48%)KM2 0.064 (3%)
HypoF TypeKM5 0.495 (86%)KM3 0.192 (13%)KM2 0.086 (1%)
HypoF AreaKM5 0.539 (74%)KM3 0.177 (3%)KM2 0.149 (22%)
HypoF FoveaKM5 0.376 (58%)KM3 0.304 (41%)KM2 0.093 (0%)
DRKM1 1.455 (100%)KM3 0.050 (0%)KM5 0.021 (0%)

We added these analyses to the revised paper (sec.6.2, Task-level and input-dependent allocation)

Q2 (Learning across very different domains): We thank the reviewer for pointing us to this work. Meyerson & Miikkulainen (2019) study knowledge sharing across tasks with different inputs and architectures (vision, language, and genomics). ASK-OT addresses a different setting: multiple tasks defined on the same input, as is common in medical imaging, where several clinical findings are assessed from the same study. Importantly, sharing an input does not imply that tasks benefit equally from shared representations. As shown in Section 3, even clinically related FA tasks exhibit asymmetric and sometimes negative transfer. Extending ASK-OT to more diverse domains would depend on how the data are structured:

Paired modalities. When multiple modalities are available for the same case (e.g., fundus images and OCT scans from the same eye), the framework could be extended using modality-specific encoders and shared knowledge modules. The transport plan could then learn how different tasks allocate these modules, potentially capturing asymmetric sharing across modalities.

Unpaired domains. For tasks such as language, vision, and DNA analysis, there may be no common input on which all tasks are defined. In this setting, modality-specific encoders would require a shared representation space for their knowledge modules to be transferable across domains. This would also require adapting the allocation mechanism: unlike our current setting, an input from one domain cannot jointly inform the allocation across tasks from other domains. Thus, while the OT-based resource allocation principle is not inherently tied to a particular domain, extending the current input-dependent sharing mechanism to unpaired domains would require additional design choices, particularly for cross-domain representation alignment and allocation.

Thanks for the answer, but I'd still like to see a high-level insight on what's been learned, in addition to the technical details. I still have just bits and pieces, no the big picture. What problem in knowledge sharing does the reseach allocation solve?

AuthorsOpenReview ↗

High-Level Insight

We thank the reviewer for the follow-up. In one sentence, resource allocation solves the problem of sharing knowledge between tasks whose relationships are asymmetric, without either forcing all tasks onto one representation or fixing in advance which tasks share with which. We summarize the big picture below; this will be added to the final manuscript.

Problem. When all tasks share one representation, their needs differ: some can benefit from overlapping information, potentially in asymmetric ways, while others need different information to be emphasized. A single representation used in the same way by all tasks cannot adequately accommodate these differences.

Our view. We build on a shared encoder that provides a common representation for all tasks, what differs between tasks is how this representation needs to be transformed and used. Knowledge modules (KMs) provide this flexibility: they are nonlinear transformations that recombine the shared representation into distinct views, each emphasizing different information (Fig. 2a). The question then becomes which tasks should use which views, and to what extent. We formulate this as a resource allocation problem: a limited set of knowledge modules must serve tasks with different representational needs.

What resource allocation solves. Simply offering multiple views is not enough, because left unconstrained they tend to become redundant. Allocating limited capacity across tasks forces the views to become distinct, and lets each task draw on them to its own degree. The sharing structure therefore emerges from the tasks’ actual needs rather than being imposed in advance.

What we learned. Shared knowledge is more effective when organized into specialized views that tasks can draw on to different degrees, rather than as a single representation used equally by all tasks. This allows knowledge sharing to reflect the actual, often asymmetric, relationships between tasks.

Reviewer_pXAXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ASK-OT, an optimal-transport-based framework for asymmetric knowledge sharing in medical multi-task learning. A shared encoder produces multiple learned self-attention knowledge modules, while each task is represented by a learned task embedding. For each input, a module-task cosine-cost matrix is converted into an entropic optimal-transport plan. A hard module-capacity constraint enforces balanced module usage, while a soft task-side constraint allows task-specific demand to vary. The transported module representations are then passed to task-specific prediction heads.

The paper evaluates ASK-OT on FA, AMD, and CelebA benchmarks using several pretrained backbones and compares it with loss-balancing, gradient-conflict, task-grouping, and architectural baselines. It reports improvements in mean relative gain and hypervolume, together with ablations for routing, module capacity, task constraints, module architectures, and transport hyperparameters. The paper’s main conceptual claim is that optimal transport can learn input-dependent, asymmetric sharing patterns without imposing predefined task groups.

Critical strengths

A meaningful formulation of asymmetric sharing. The paper goes beyond fixed task groups and explicitly separates module supply from task demand. This is a useful design perspective for heterogeneous multi-task learning, where symmetric parameter sharing is often too restrictive.

Reasonable component-level evaluation. The paper includes routing, module architecture, MCC, TCR, module-count, and temperature ablations, and evaluates multiple datasets and pretrained backbones. The patient-level split on FA is also a positive aspect of the experimental protocol.

Critical weaknesses

The central knowledge-sharing interpretation is confounded by unnormalized transport mass. The task representation is defined as zn′=∑mΠmnzmz'_n=\sum_m \Pi_{mn}z_m, without normalizing by the task’s total mass an=∑mΠmna_n=\sum_m\Pi_{mn}. Because the task-side TCR constraint intentionally allows ana_n to vary, ASK-OT can change the magnitude of each task representation and therefore its effective loss or gradient scale. The reported gains may therefore come from adaptive task-wise rescaling rather than from asymmetric module sharing. The paper does not provide the essential control in which task representations are normalized by ana_n, or in which all task column masses are fixed.

The empirical evidence is too fragile and the comparisons are not fully matched. ASK-OT introduces additional self-attention modules, task embeddings, and routing parameters, but the main comparisons do not appear to use parameter- or FLOP-matched adaptive architecture controls. AMD contains only about 400 images and exhibits large variance, while CelebA-40 is mainly compared against GD with only three seeds. No paired per-task statistical tests or confidence intervals are provided. The theoretical statements also rely on assumptions such as task embeddings being faithful proxies for task requirements, but these assumptions are not empirically validated.

Questions for authors

The task representation is defined as zn′=∑mΠmnzmz'_n=\sum_m \Pi_{mn}z_m, while the soft TCR constraint allows the total task mass an=∑mΠmna_n=\sum_m\Pi_{mn} to vary. Please report a matched control that either normalizes zn′z'_n by ana_n or fixes all task column masses. Do the per-task gains and the claimed asymmetric-sharing behavior remain under this control?

Please provide parameter-, FLOP-, and activation-memory-matched comparisons against an adaptive gating or mixture-of-experts baseline without optimal transport. How much of the reported improvement remains after accounting for the additional knowledge modules, task embeddings, and routing computation?

Please report per-task and aggregate results over multiple random seeds, including confidence intervals or paired significance tests, especially on AMD. Also specify the exact normalization and reference set used for hypervolume, since the current aggregate summaries may conceal substantial task-level regressions.

5 comments

We thank the reviewer for the constructive reviews. Below, we address questions, which have been incorporated into the revised paper.

Q1. Task column normalization: This question helped us clarify an important aspect of ASK-OT. We conducted both requested control experiments (i.e. task column normalization and fixing), keeping all other settings unchanged .

Before presenting the results, we would like to clarify an important connection between the column mass ana_n and asymmetric sharing. Under TCR, ana_n allows each task to express how much it needs from the shared modules. Thus, it is not independent of asymmetric sharing; rather, it is one of the mechanisms through which asymmetric sharing is realized.

Normalizing zn′z'_n by ana_n. Following the reviewer's suggestion, we normalized the task representation after computing the optimal transport solution: zn′=∑m(Πmn/an)zmz'_ n = \sum_m (\Pi_{mn}/a_n) z_m. Although this leaves the OT solution unchanged, it changes how the resulting assignments influence module updates. Specifically, it rescales each task's contribution from Πmn\Pi_{mn} to Πmn/an\Pi_{mn}/a_n, effectively fixing the total contribution of every task across modules to one, regardless of its demand. Consequently, tasks with high overall demand have their influence reduced, while tasks with low overall demand gain relatively more influence on module updates. This also counteracts the purpose of TCR: tasks are allowed to express different levels of demand through ana_n, but the subsequent normalization removes this distinction from their contributions to module updates. Finally, because normalization is applied after OT, the resulting effective assignments no longer satisfy the MCC, which is essential to controlling the allocation of shared modules. For these reasons, we expect decrease of performance by this normalization.

Results. Overall performance decreases on both datasets: the mean AP drops by 1.39 on FA and 1.11 for classification tasks on AMD, and the fovea localization MAE on AMD increases by 0.03. The validation-averaged transport plans of the normalized model still show the asymmetric patterns.

Fixed task column masses. Fixing all column masses corresponds to enforcing the task-side constraint as a hard equality, which is the OT+MCC variant of our ablation (Table 3). In this setting, every task receives the same total mass, so all zn′z'_n have the same scale. This variant reaches +2.09% on FA and +7.78% on AMD, compared with +4.46% and +13.07% for the full model. Note, however, that fixing the column masses also changes the transport plan itself, since the routing must then satisfy the hard constraint.

Interpretation. The two controls remove the flexibility of ana_n in complementary ways: after OT in the first case, and within OT in the second. Both lead to lower performance than the full model. Together, these results suggest that the column mass ana_n plays a role in ASK-OT.

Table: Effect of normalizing the task representation by its column mass (mean ± std over 5 runs). Task rows report AP (↑), except fovea localization error (↓). Diff.: Normalized − ASK-OT. Mean AP is computed over classification tasks.

TasksASK-OTNormalizedDiff.
FA dataset
DR47.95 ± 3.7144.63 ± 2.74-3.32
CNV85.62 ± 2.9785.22 ± 0.57-0.40
HyperF Type52.53 ± 2.0153.09 ± 1.33+0.56
HyperF Fovea83.07 ± 2.2383.81 ± 1.44+0.74
HyperF Area78.73 ± 1.5777.55 ± 0.68-1.18
HypoF Type65.07 ± 3.9660.79 ± 2.48-4.27
HypoF Fovea49.91 ± 2.9446.59 ± 1.84-3.32
HypoF Area61.81 ± 2.7861.90 ± 2.29+0.08
Mean AP65.5964.20-1.39
AMD dataset
AMD84.32 ± 4.6185.94 ± 5.74+1.62
Fovea localization (↓)3.35 ± 0.053.38 ± 0.15+0.03
Drusen76.54 ± 6.6375.51 ± 9.97-1.03
Exudate80.35 ± 6.2870.45 ± 14.13-9.91
Hemorrhage25.53 ± 7.0530.59 ± 14.70+5.06
Scar65.15 ± 9.2763.87 ± 4.94-1.28
Mean AP66.3865.27-1.11

Q2. Comparison with Adaptive Gating and Computational Cost: We address this through our ablation study (Table 3) and also added an adaptive gating variant to this ablation as suggested by the reviewer. All variants share the same encoder, knowledge modules and decoders, and differ only in how module outputs are routed to tasks.

(1) Additional knowledge modules: The "KM + Uniform" row adds the same 5 KMs to the GD, but combines them with fixed uniform weights, without task embeddings or any routing. It therefore contains all parameters ASK-OT adds, except the task embeddings (N×dN \times d, negligible). This variant gives -0.67% on FA and +5.30% on AMD, compared with +4.46% and +13.07% for ASK-OT. Additional capacity thus explains none of the gain on FA and less than half of it on AMD.

(2) Softmax: The Softmax row keeps every component of ASK-OT, including KMs, task embeddings and cost matrix, and replaces only the OT solver. It normalizes the cost matrix over modules for each task, so every task spreads its routing weight across modules, with lower-cost modules receiving more weight. This corresponds to a transport plan in which only the task-side marginal is enforced. Since OT introduces no learnable parameters, this variant has exactly the same parameter count as ASK-OT. It reaches +2.21% (FA) and +11.33% (AMD). The OT formulation with MCC and TCR therefore adds +2.25 and +1.74 over softmax routing with identical architecture.

(3) Adaptive gating: We add a learned, input-conditioned gate, built on the same KMs. For each input and each task, a small linear gate takes a summary of the KMs outputs together with a learned task-specific term, and produces a softmax distribution over the KMs. These weights then combine the module outputs into the task representation. Unlike the Softmax variant, the routing is learned by dedicated gate parameters rather than derived from a module-task cost. Here, the experts are our attention modules, so the architecture matches ASK-OT. It obtains -0.94% on FA and +10.30% on AMD, below both Softmax and ASK-OT. ASK-OT outperforms it by +5.40 (FA) and +2.77 (AMD).

Parameters, FLOPs and activation memory: The following table reports the computational cost of the four variants on the FA dataset (224 × 224 inputs). Parameters are the trainable parameters of the full model. FLOPs are forward-pass GFLOPs per image, measured with PyTorch's built-in FlopCounterMode. Following the standard convention, it counts matrix multiplications, convolutions and attention, but not element-wise operations. Activation memory is the total size of the tensors autograd stores for the backward pass in one training-mode forward pass of a single image. Wall-clock times are measured on a single NVIDIA RTX 3090.

Table: Computational cost on the FA dataset.

RoutingParams (M)GFLOPs / imageAct. memory (MB / image)Step time (ms)
Uniform24.59913.262111.06–
Softmax24.60213.262111.09139.0
Adaptive gating24.60313.262111.09139.0
ASK-OT24.60213.262111.13170.4

The four variants are matched in parameters (within 0.02%), FLOPs, and activation memory (within 0.07 MB per image, about 0.06%). This is expected, because they differ only in how the M×N routing weights are computed. Since the FLOP counter excludes element-wise operations, the Sinkhorn iterations are not included in the reported FLOPs. Each iteration performs two log-sum-exp operations over the 5×8 cost matrix, roughly 4×10410^{4} FLOPs per image, about 0.0003% of the forward pass, which does not change the comparison. The only measurable overhead of OT is wall-clock time. The Sinkhorn iterations run sequentially, which adds about 31 ms per training step (about 23%). This is consistent with Figure 6, where ASK-OT remains close to GD and well below most baselines in epoch time. The overhead could be reduced further with fewer iterations or a convergence-based stopping criterion.

Summary. After accounting for the additional KMs, task embeddings and routing, the improvement attributable to the OT formulation is +2.25/ +1.74 over softmax and +5.40 /+2.77 over learned adaptive gating (FA / AMD), under matched parameters, FLOPs and activation memory.

Q3. HV definition, HV with GD as reference point:

Hypervolume definition. For each method, the per-task test scores of one seed form one point: AP for the classification tasks and 1-MAE for fovea localization (computed over the two normalized coordinates). Every coordinate therefore lies in [0,1]. HV is the volume of the region dominated by their Pareto front, measured from the fixed reference corner (0,..,0), i.e., the worst possible score on every task.

Joint improvement over GD, and HVc=GDHV_{c=GD}. We agree with the reviewer that aggregate summaries can conceal task-level regressions. The HV above rewards joint performance, since its volume is a product over tasks, but its reference point is the worst possible score, so a task that falls below GD still contributes volume. To measure joint improvement over the baseline directly, we additionally report HVc=GDHV_{c=GD}: the same hypervolume, with each task as one axis and the reference point moved from the worst possible score to a level relative to GD. Each axis measures the gain over GD on one task, and the volume is the product of these gains.

For each task, we compute the relative gain of the method's seed-mean over the seed-mean of GD, so that positive relative gain means better than GD. We place the reference point at (-c,..,-c), where c is the largest relative loss with respect to GD that a task may have and still contribute to the volume (e.g., c=25%GD means a task may be at most 25% worse than GD). The hypervolume is divided by cNc^N so that GD itself achieves exactly 1, so a value above 1 indicates joint improvement over GD. Because the volume is a product over tasks, a regression on any task shrinks it. We use one point per method rather than one per seed so that seed-to-seed variation cannot inflate the score. We report c=25% as the strict setting, in which a method must stay within 25% of GD on every task to obtain any volume, and c=100% as a lenient setting that checks whether the ranking depends on this choice.

Table: HV with c = % of GD.

MethodFA (N=8), c=25%FA (N=8), c=100%AMD (N=6), c=25%AMD (N=6), c=100%
GD1111
FAMO2.011.210.000.40
Nash-MTL2.391.260.000.84
CAGrad1.741.181.911.77
Ours2.921.392.391.88

Results. ASK-OT obtains the highest HVc=GDHV_{c=GD} on both datasets and at both reference gains. On FA, it reaches 2.92 at c=25%, compared with 2.39 for Nash-MTL (1.22×) and 2.01 for FAMO (1.45×), and 1.39 at c=100%, compared with 1.26 and 1.21. On AMD, it reaches 2.39 at c=25%, compared with 1.91 for CAGrad (1.25×), and 1.88 vs. 1.77 at c=100%. Since the margin over the strongest baselines holds under both settings, the gains of ASK-OT reflect joint improvement across tasks.

Q3.(per-task and aggregate significance)

With 5 seeds, paired tests cannot reach significance (the Wilcoxon test cannot go below p=0.0625), and bootstrap CIs are too coarse to be informative. So, we extended the evaluation to 20 seeds (seeds 1-20) for ASK-OT, and the strongest baselines for each dataset, and compared ASK-OT with Nash-MTL (FA) and CAGrad (AMD). For Δ and for each task, we report paired t/Wilcoxon tests and paired bootstrap 95% CIs (10,000 resamples).

Aggregate results. In terms of Δ, ASK-OT improves significantly over both Nash-MTL on FA and over CAGrad on AMD. It does so at a considerably lower cost i.e., about one fifth of their training time (Fig.6c). In terms of HV, ASK-OT is higher than both baselines, the difference is significant on AMD.

On FA, Δ improvement of ASK-OT over Nash-MTL is significant: the difference is +1.65 (CI [+0.26,+3.06];t-test p=0.036,Wilcoxon p=0.040). Its HV is also higher (0.0467 vs 0.0399), but this difference is not significant. On AMD, ASK-OT's Δ is higher than CAGrad's by +5.84 on average, and the difference is statistically significant (CI [+0.60,+10.48]; t-test p=0.035,Wilcoxon p=0.009). HV is also significantly higher for ASK-OT (0.370 vs 0.136;p<0.001).

FA vs Nash-MTLAMD vs CAGrad
Δ difference [95% CI]+1.65 [+0.26, +3.06]+5.84 [+0.60, +10.48]
Δ: paired t / Wilcoxon p0.036 / 0.0400.035 / 0.009
HV ours / baseline0.0467 / 0.03990.3700 / 0.1361
HV: permutation p (two-sided)0.199<0.001

Per-task results. On FA, ASK-OT significantly improves DR (+5.18) and HypoF Fovea (+2.71), shows a small but significant drop on HypoF Area (-1.79), and is statistically on par on the remaining five. On AMD, ASK-OT is significantly better on hemorrhage (+11.28) and exudate (+9.52), with no significant difference on drusen and scar. CAGrad is significantly better on AMD diagnosis (-3.50) and fovea localization (-0.40).

TaskASK-OTBaselineDiff. [95% CI]p
FA: ASK-OT vs Nash-MTL (AP, ↑)
DR46.0240.84+5.18 [+2.70, +7.83]<0.001
CNV85.2884.78+0.50 [-0.58, +1.53]0.370
HyperF Type52.7153.18-0.47 [-1.64, +0.65]0.442
HyperF Fovea83.8883.13+0.74 [-0.54, +1.99]0.278
HyperF Area78.8079.18-0.38 [-1.25, +0.52]0.432
HypoF Type63.4265.32-1.91 [-4.04, +0.35]0.112
HypoF Fovea48.3245.61+2.71 [+1.04, +4.33]0.005
HypoF Area62.2964.08-1.79 [-3.19, -0.37]0.026
AMD: ASK-OT vs CAGrad (AP, ↑; fovea: MAE, ↓)
AMD84.3587.85-3.50 [-5.93, -1.25]0.011
Fovea localization3.392.99-0.40 [-0.53, -0.26]<0.001
Drusen73.3374.37-1.04 [-6.42, +4.61]0.722
Exudate74.3464.82+9.52 [+4.13, +14.72]0.003
Hemorrhage35.0423.76+11.28 [+4.39, +18.28]0.006
Scar59.6259.14+0.48 [-7.83, +8.86]0.911
*Diff. is oriented so that positive means ASK-OT is better.

We added these analyses to the Appendix G.

Reviewer_xc4gOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ASK-OT casts representation-level MTL sharing as an OT resource-allocation problem. A shared encoder feeds learnable nonlinear knowledge modules; task embeddings define module–task affinities; a Sinkhorn plan Π⋆ mixes module outputs into per-task representations (Sec. 4; Fig. 2). Balanced OT is modified with (i) Module-Capacity Constraint (MCC), scaling each module's row mass to N/M to encourage spillover (Eq. 3; Lemma 1), and (ii) Task Constraint Relaxation (TCR), a soft KL on the task-side marginal that allows uneven total allocation (Eq. 4; Lemma 2). Asymmetric sharing is motivated by a small FA transfer study (Sec. 3; Fig. 1; App. A/Table 4). Theory covers forced spillover and directional asymmetry (Sec. 5; Apps. C–D). Experiments: FA (8 tasks), AMD (6 tasks), CelebA-9, CelebA-40 vs loss-balancing, gradient-conflict, grouping, and architecture baselines (Tables 1–3; Figs. 3–7). Aggregate Δ and HV favor ASK-OT on the medical sets; ablations attribute gains to MCC+TCR over uniform/softmax/balanced-OT routing.

Critical strengths

1. Asymmetric sharing motivation matches the OT design. Sec. 3 / Fig. 1 (and Table 4) show clinically related FA tasks transfer asymmetrically and that no fixed grouping is jointly optimal. Learnable modules + joint transport under MCC (Eq. 3) and TCR (Eq. 4) fit that problem; Lemmas 1–2 / Proposition 1 formalize how capacity and soft task margins induce non-symmetric directional mass (Sec. 5). This problem–method fit is the strongest reason for acceptance.

2. Broad evaluation with ablations. Tables 1–2 and Fig. 3 cover medical in-/out-of-domain settings and CelebA scale; baselines include STL, GD, FAMO, Nash-MTL, IMTL, STCH, CAGrad, FairGrad, selTG, MMoE, and TSσLN. Table 3 isolates KM, Softmax, balanced OT, MCC, and TCR; Figs. 5–6 and App. F support specialization, soft (not collapsed) routing, and modest compute overhead. HV on AMD (0.162 vs CAGrad 0.110) complements Δ, so gains are not only isolated per-task wins.

Critical weaknesses

1. Incremental vs MoE-style routing; OT necessity only partly shown. ASK-OT is close to a shared-expert / soft-gated bank (cf. MMoE in Sec. 2). Same-KM Softmax already reaches +2.21% (FA) / +11.33% (AMD) vs ASK-OT +4.46% / +13.07% (Table 3), so much of the AMD gain looks like modules + independent routing rather than capacity-coupled OT. Theory largely follows from marginal constraints and unbalanced-OT KKT forms (App. C) and does not give performance or negative-transfer guarantees. Without stronger evidence that a single joint plan is essential beyond Softmax, the conceptual advance is limited.

2. Medical headline claims are statistically fragile. FA (~1.7k) and AMD (400 images; App. E) are small; Table 1 has large stds (e.g., AMD scar 65.15±9.27, hemorrhage 25.53±7.05). AMD Δ is only modestly above CAGrad (+13.07 vs +12.15) despite a larger HV gap; several tasks do not favor ASK-OT (FA HyperF Fovea; AMD diagnosis and fovea localization). CelebA-40 compares mainly to GD, and τ,M are untuned there (Table 5). Without significance tests, CIs on Δ/HV, or stronger matched-capacity controls, "outperforms representative baselines" is not firmly established.

Questions for authors
  1. OT vs Softmax. On AMD, Softmax is within ~1.7 Δ of full ASK-OT (Table 3). Where does independent Softmax fail because it cannot enforce MCC-style capacity coupling (task counts, conflict regimes, held-out combinations), and how large is that gap under matched parameter/FLOP budgets?

  2. Significance. For Table 1, do paired tests or bootstrap CIs over the 5 seeds show ASK-OT's Δ and HV significantly above Nash-MTL (FA) and CAGrad (AMD)? How sensitive is the ranking to using GD as the Δ reference?

  3. Π(x) vs task-level sharing. C_{m,n}=1−sim(z_m(x), t_n) makes the plan sample-dependent (Alg. 1–2). Is the claim about a stable task-sharing structure, or per-image soft routing? Does a frozen validation-average plan (Figs. 8–9) keep most of the gain?

  4. Orthogonality. Have you combined ASK-OT routing with FAMO/CAGrad/Nash-MTL? Non-additive gains would change how I weigh representation-level vs optimization-level contributions.

(Q1–Q3 could change my rating; Q4 matters for positioning but is secondary.)

2 comments
AuthorsOpenReview ↗

Part 1/2 (Q1 (OT vs. Softmax))

We thank the reviewer for the constructive feedback and reviews. Below, we address questions, which have been incorporated into the revised manuscript.

Q1 (OT vs. Softmax): Matched budget. The Softmax and ASK-OT variants in Table 3 share the same knowledge modules, task embeddings and decoders, so their parameter counts are identical. The only difference is the routing step. ASK-OT runs 100 Sinkhorn iterations on a small M×N matrix (5×8 on FA), which adds negligible cost. The comparison is therefore already matched in parameters and nearly matched in compute. Under this budget, ASK-OT improves over Softmax by +2.25/+1.74 on FA/AMD.

The following table reports the computational cost of both variants on the FA dataset (224×224224 \times 224 inputs). Parameters are the trainable parameters of the full model. FLOPs are forward-pass GFLOPs per image, measured with PyTorch's built-in FlopCounterMode, counting a multiply-add as 2 FLOPs. Following the standard convention, it counts matrix multiplications, convolutions and attention, but not element-wise operations. Activation memory is the total size of the tensors autograd stores for the backward pass in one training-mode forward pass of a single image. Wall-clock times are measured on a single NVIDIA RTX 3090.

RoutingParams (M)GFLOPs / imageAct. memory (MB / image)Step time (ms)
Softmax24.60213.262111.09139.0
ASK-OT24.60213.262111.13170.4

Both variants are matched in parameters, FLOPs, and activation memory (within 0.04 MB per image). Since the FLOP counter excludes element-wise operations, the Sinkhorn iterations are not included in the reported FLOPs. Each iteration performs two log-sum-exp operations over the 5×85 \times 8 cost matrix, roughly 4×1044 \times 10^{4} FLOPs per image, about 0.0003%0.0003\% of the forward pass, which does not change the comparison.

The only measurable overhead of OT is wall-clock time. The 100 Sinkhorn iterations run sequentially, which adds about 31 ms per training step (170.4 vs. 139.0 ms, about 23%). This is consistent with Figure 6, where ASK-OT remains close to GD and well below most baselines in epoch time. The overhead could be reduced further with fewer iterations or a convergence-based stopping criterion.

What differs. Softmax normalizes the cost matrix over modules for each task, so every task spreads its routing weight across modules, with lower-cost modules receiving more weight. This corresponds to a transport plan in which only the task-side marginal is enforced: each task selects its modules independently, and nothing limits how much weight each module receives in total. ASK-OT adds a hard module-side constraint that couples the tasks through shared module capacity.

Where Softmax falls short. To examine the effect of this coupling, we tracked the load of each module during training, i.e., the total routing mass it receives. This load is the row sum of the transport plan averaged over the validation set, Loadm=∑nΠˉmnLoad_m=\sum_n \bar{\Pi}_{mn}. We summarize how evenly it is spread with the coefficient of variation across modules, CoV=stdm(Loadm)/meanm(Loadm)\mathrm{CoV}={\mathrm{std}_m(Load_m)}/{\mathrm{mean}_m(Load_m)}, where 0 corresponds to a perfectly even load. Under Softmax, the load remains uneven throughout training and stabilizes at CoV≈0.27\mathrm{CoV}\approx0.27 on FA and 0.26 on AMD. Under ASK-OT, MCC keeps every load at N/M, so CoV=0\mathrm{CoV}=0 by construction.

This suggests that, under Softmax, tasks tend to concentrate on some modules. We also observe that module output similarity is comparable under both methods, indicating that Softmax does learn distinct modules. The difference therefore appears to lie mainly in how tasks are allocated to these modules, which is the aspect MCC is designed to control. We added this analysis in the revised paper(Appendix F. Routing Variants: Analysis and Computational Cost).

AuthorsOpenReview ↗

Part 2/2 (Q2 (Significance) and Q3 ( vs. task-level sharing))

Q2

With 5 seeds, paired tests cannot reach significance (the Wilcoxon test cannot go below p=0.0625), and bootstrap CIs are too coarse to be informative. So, we extended the evaluation to 20 seeds (seeds 1-20) for ASK-OT, and the two baselines named by the reviewer. For Δ, we use paired tt/Wilcoxon tests and paired bootstrap 95% CIs (10,000 resamples).

Overall: In terms of Δ, ASK-OT improves significantly over both baselines. It does so at a considerably lower cost: about one fifth of their training time (Fig.6c). In terms of HV, ASK-OT is higher than both baselines, the difference is significant on AMD.

FA. Δ improvement of ASK-OT over Nash-MTL is significant: the difference is +1.65 (CI [+0.26,+3.06];t-test p=0.036,Wilcoxon p=0.040). Its HV is also higher (0.0467 vs 0.0399), but this difference is not significant.

AMD. ASK-OT's Δ is higher than CAGrad's by +5.84 on average, and the difference is statistically significant (CI [+0.60,+10.48]; t-test p=0.035,Wilcoxon p=0.009). HV is also significantly higher for ASK-OT (0.370 vs 0.136;p<0.001). Added to Appendix G.

FA vs Nash-MTLAMD vs CAGrad
Δ difference [95% CI]+1.65 [+0.26, +3.06]+5.84 [+0.60, +10.48]
Δ: paired t / Wilcoxon p0.036 / 0.0400.035 / 0.009
HV ours / baseline0.0467 / 0.03990.3700 / 0.1361
HV: permutation p (two-sided)0.199<0.001

Sensitivity to GD reference. Using Table 1, we recomputed Δ with each of the 11 MTL methods (including ASK-OT itself) and the mean of all methods as the reference. On FA, ASK-OT remains first in all 12 cases and on AMD, ASK-OT remains first in 11/12. The one exception is FairGrad as the reference, where CAGrad is ahead by a small margin of 0.42. The picture is consistent across reference choices: ASK-OT ranks first under every reference on FA and under all but one on AMD, in agreement with its significant Δ improvements over Nash-MTL/CAGrad on FA/AMD.

Q3

Since the cost depends on the module outputs zm(x)z_m(x), the transport plan is indeed computed per input. The learned sharing can therefore be viewed as having two components: a task-level structure common to all inputs, captured by the average plan Πˉ=Ex[Π(x)]\bar{\Pi} = \mathbb{E}_x[\Pi(x)], and input-specific adjustments around it. To assess the role of each component, we tested whether Πˉ\bar{\Pi} can replace the per-input plans, as suggested by reviewer.

Effect on performance. Replacing Π(x)\Pi(x) with its validation average at test time reduces the gain over GD on both datasets, from +4.46% to +0.68% on FA and from +13.07% to +10.15% on AMD. This indicates that the averaged plan captures meaningful task-level structure, but per-input allocation is required to obtain the full performance gain. We hypothesize that these adjustments reflect variation within each task: since the task embeddings are shared across all images, the plan is organized primarily around tasks, while images from different subgroups of the same task (e.g., different lesion patterns under the same task) may draw on the modules to different degrees.

Qualitative case study: input-dependent allocation within a task. As a qualitative example of this hypothesis, we visualize the HyperF Type task on FA, whose five classes correspond to visually distinct patterns. Table 4 in the revised paper summarizes the task-level structure. For HyperF Type, the plan concentrates on KM4 and KM3, followed by KM2. Figure 8 in the revised paper shows, for one test image per class, the attention map of each KM and corresponding per-input plan weights.

The same modules are used, but to different degrees: For all abnormal classes, the images rely mainly on the modules favored at the task level, but the degree of reliance varies with the abnormality type, and the attention maps of modules focus on the abnormal regions. This shows the decomposition described above: Πˉ\bar{\Pi} identifies the set of relevant modules, while the per-input plan adjusts how strongly each image relies on them. Normal image, which has no focal lesion, is handled differently. The allocation is spread over several modules, and the attention maps cover broad areas of the retina rather than a single region. We interpret this as a learned strategy: confirming the absence of hyperfluorescence requires inspecting the whole image. Overall, this study is consistent with our hypothesis: ASK-OT appears to learn a task-level sharing structure, with per-input allocation reflecting variation within a task. This offers a possible explanation for why freezing the plan to Πˉ\bar{\Pi} retains part of the gain but not all of it. Added to Sec.6.2

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.