acceptodds
Under review as a conference paper at ICLR 2027

FedM3G: Federated Multimodal Multi-task Graph Learning

Abstract

Federated learning over multimodal attributed graphs lets clients with complementary modalities and heterogeneous downstream tasks collaborate without exposing private data. Under the emerging public-dataset paradigm, clients exchange representations computed on a shared graph rather than model parameters. However, we identify two fundamental obstacles overlooked by current federated multi-task and graph learning methods. First, prevailing multimodal graph encoders entangle node-intrinsic semantics with neighbor-derived topology, so that representation alignment indiscriminately mixes two heterogeneous information streams. Second, task-agnostic alignment forces representations from unrelated tasks onto a single manifold, triggering severe negative transfer. To address these challenges, we propose FedM3G, a federated multimodal multi-task graph learning framework that decouples and selectively aligns representations. On the client side, Gram–Schmidt semantic–topological decoupling exposes the semantics-orthogonal residual of neighbor context and softly combines it with task-relevant aggregated information, yielding a semantic track and a topology-oriented track without imposing rigid orthogonality on their final outputs. On the server side, Task Kernel Alignment estimates label-free public-probe compatibility between clients, and personalized dual-track synergistic representation alignment converts the scores into client-specific teachers and confidence-weighted objectives, with sparse curriculum masking suppressing weak peer relations. Across different datasets, FedM3G consistently outperforms the strongest baseline, with absolute gains reaching up to 16.21% on cross-modal alignment and 4.99% to 9.21% under joint task and domain heterogeneity across five downstream tasks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Federated Graph Learning, Multi-task Learning, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nCKjOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
3(+2)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies task heterogeneity in multimodal federated learning, where the main challenge is to align cross-client knowledge while avoiding interference from misaligned local task information. The proposed method decouples alignment into semantic and topological components. The server employs a kernel-based alignment method to estimate knowledge compatibility between clients, and the resulting correlation scores are used to construct a personalized teacher for each client. While the problem setting is practical and challenging, and the proposed method is tailored to address it, several concerns remain that may affect the persuasiveness and contribution of this paper.

Critical strengths
  1. The problem studied is interesting and practical. Local task structures can significantly affect the knowledge extracted from multimodal data. Moreover, the label-free setting reflects a classic compatibility challenge in typical federated learning scenarios.

  2. The proposed method is well-described and well-aligned with the target problem, making the problem more tractable. Especially, the TKA design is reasonable and computationally efficient. It compares the relational geometry of representations as a label-free proxy.

  3. The experimental evaluation appears comprehensive, covering 9 baselines, 3 datasets, multiple backbones, and a Q2 stress test with mixed domains and tasks. Using Fig. 1 as a preliminary experiment to motivate the proposed method is also appropriate.

Critical weaknesses
  1. The metric Align CLIP only reports the alignment score of positive pairs, what about the negative pairs? Is the improvement is from the learn contrastive information or just collapsed results (especially considering the score raises up to 0.99 on some baselines) ? The improvement of the alignment score is already noticeable without the designed SRA module, which is basically how the clients share knowledge between each other, removing it means that the clients are only doing local training only. Also, the local training baseline should be included in ablation study and main comparison.

  2. The ablation study shows that removing TKA leads to lower performance on some metrics than removing SRA. This suggests that incompatible knowledge sharing could perform worse than no sharing. I think further analysis is needed to better understand this phenomenon, such as examining the training curves and investigating how informative the uniformly constructed representations are for local tasks. The interactions between different modules are not sufficiently studied. It would be helpful to replace some modules with simpler designs or components adopted from existing baselines to better understand the contribution of each module compared with simpler alternatives.

  3. In the ablation study, the sweep of λ\lambda covers the range [0.01, 0.5], a 50-fold range, yet the reported metrics barely change across this range. In contrast, Table 3 shows substantial performance changes when SRA is removed. Although the two experiments use different datasets, these results seem inconsistent w.r.t. the claim. Would a wider range of λ\lambda be needed to better understand its effect, or would it be more appropriate to use the same dataset for both experiments?

  4. In Table 2, FedAvg and FedLPS have exact same results (even the std.) on some metrics.

  5. A highly relevant benchmark work "MM-OpenFGL: A Comprehensive Benchmark for Multimodal Federated Graph Learning" is not cited or discussed in this paper.

  6. The paper states that "CreamFL fails to converge on LinkPred on all three datasets because its prediction-level distillation on the public batch ..." However, to the best of my knowledge, CreamFL communicates per-sample representations of public data (a contrastive representation ensemble), not predictions or logits. Could the authors clarify this explanation?

  7. A reproducibility section should be included in the appendix to facilitate the reproduction of the results for both the proposed method and the baselines.

  8. Potentially incorrect references, possibly hallucinated by an LLM. The following references should be verified:

  • GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner (author list and venue)
  • Graph Structure Mapping Neural Networks for Generative Graph Self-Supervised Learning
Questions for authors

See the weaknesses above.

4 comments

We appreciate your recognition of the practical problem, the label-free compatibility design, and the breadth of our experimental evaluation.

W1: We appreciate your question. In the experiments presented in the main table, the cosine similarity between positive pairs is significantly higher than that between negative pairs. In this implementation, disabling SRA removes the representation-alignment loss while maintaining the communication loop and the server-provided prototype terms. The "Local-only" setting further disables communication and public-data updates, while also clearing all server-provided guidance and prototypes. Since there is only one client per task, task-grouped prototypes may originate from that same client; therefore, the mere retention of the communication loop does not constitute cross-client transfer. If all server-dependent updates are disabled, the training process effectively reduces to the "Local-only" setting. We will include Local train as a baseline in the main text.

W2: Removing TKA retains cross-client alignment but removes compatibility-based peer selection. Incompatible guidance can therefore compete with the local task objective, whereas removing SRA removes that alignment pressure. This provides a mechanism through which unselective sharing can perform worse than disabling alignment.

The semantic-only and topology-only rows in Table 3 show different preferences, with semantic features favoring retrieval and matching and topology-oriented features favoring link prediction. These comparisons motivate separating the tracks before selecting transfer targets. They assess track contribution, while peer-selection effects are addressed by the uniform-weight controls.

We also tested uniform peer weights and replaced GS gating/projection with direct self/aggregate features. In the strict comparison, retaining GS improves Retrieval under both CKA and uniform peer weights.

W3: Thank you very much for your suggestion. We examined the wider range of settings you proposed. As λ\lambda increases from 1 to 10, the MR drops from 57.21% to 33.86%; this change becomes even more pronounced at higher weight values. This pattern suggests that an excessively strong alignment constraint may interfere with the objectives of local tasks. However, the range where such significant fluctuations occur falls outside the scope of typical hyperparameter selection.

W4: Thank you for highlighting the identical FedAvg/FedLPS results. We rerun the experiment, verified the execution logs and reproduces the reported means and standard deviations. A plausible explanation is that the two methods reduce to the same effective local prediction process for these tasks under the adopted backbone and task-head interface. FedAvg and FedLPS differ in how shared parameters are updated and constrained. Identical results arise when the differences between methods do not affect the effective prediction pathway of a task. Our controlled reruns support this explanation. Before and after activating FedLPS freezing and pruning changes graph-task performance while leaving the three modal tasks unchanged.

W5: Thank you for pointing out this relevant benchmark. We will discuss MM-OpenFGL as the broader benchmark context.

W6: We would like to clarify the communication object and the scope of our comparison. Our implementation uploads per-sample public representations and trains the server against aggregated representation targets using a representation-level MSE loss. This is consistent with the representation-based communication described in CreamFL). We will replace “prediction-level distillation” with “representation-level distillation” to accurately describe the implemented operation.

The relevant distinction for LinkPred concerns the learning objective. Matching representations on public nodes does not directly optimize the scores of positive and negative edges on a client's private graph. Edge discrimination also depends on the graph encoder, edge-scoring head, local supervision, and interaction with the shared objective. This distinction explains why public representation matching alone does not guarantee improved LinkPred performance. It does not, by itself, establish the cause of the observed low AUC.

W7: We appreciate your recommendations for clearer configuration and reproducibility reporting. We will add a table in appendix covering more detailed information.

W8: To be honest, we did not actually use these two backbones in our experiments; they were simply included during our literature review and data collection process. Removing these two references will not affect the readability of the paper. Regarding the incorrect references, we fully acknowledge the mistak that we copied the wrong citation sources and failed to provide the references in the correct manner.

Reviewer_nCKjOpenReview ↗

Follow-up on the authors' response

Thank you for the response and the revision. Some of my concerns have been addressed, and I will raise my score to 2 for now. The following issues remain:

  1. Incorrect references. The reference list still contains errors. Fallah et al. (2020) is titled "Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach", not "A MARS approach". The third author of Wang et al. (CVPR 2019) is Barnabás Póczos, not "Slawomir Pociot". Please check every reference against its published version.

  2. Identical ablation results. In Table 3, "w/o TKA" and "w/o GS decoupling" are identical on NodeCls (43.85±0.1) and Align CLIP (83.21±0.6) and within 0.02 on Match; "w/o SRA" gives Align CLIP 83.23±0.6. Three different interventions producing the same value suggests these metrics are governed by something other than the removed module. Please confirm that these rows come from separate runs and explain why the values coincide.

  3. Effect of TKA. With five clients (one per task, Table 4) and k0 = 3 (Table 6), each client has four peers, so the mask excludes at most one peer early in training and none once the curriculum ends. The only other difference from the uniform-weight ablation is the softmax reweighting. Please show the CKA matrices and the resulting peer weights over rounds, and explain how this produces an 8.64-point LinkPred drop.

  4. λ sensitivity. The new sweep only extends upward (λ from 1 to 10), but the inconsistency I raised is at the lower end. Table 3 implies that removing the SRA loss (λ = 0) costs 6.9 (Align), 4.7 (Retrieval) and 7.75 (LinkPred) points, while Fig. 4 reports a 0.03% macro drift over [0.01, 0.5]. Please add λ = 0 and one or two values below 0.01 to the sweep, and run the ablation and the sweep on the same dataset and backbone.

We sincerely appreciate the reviewer’s continued engagement and insightful guidance. Your thoughtful reassessment and constructive suggestions have been invaluable in refining our discussion and improving the clarity and rigor of the manuscript.

Q1: We apologize for the inaccuracies in the references; both issues you identified are indeed correct. We will rectify these entries and conduct a thorough, item-by-item review against the original publication records. Furthermore, we will consult the original papers to verify that each citation accurately supports the corresponding claims in the text. We will also correct any imprecise descriptions of methodology and any attributions that lack sufficient evidence, incorporating these revisions into the updated manuscript.

Q2: I have reviewed the relevant experimental logs and confirmed that these results were obtained from separate runs. In fact, the raw values ​​were not identical; for the sake of clear presentation, the figures in the paper have been rounded. Removing different components affects distinct stages of collaboration but may lead to similar losses in collaborative gains for the same downstream task. For example, all three ablated variants achieve Align CLIP scores of approximately 83.2, about 6.9 points below the full model. This pattern is consistent with cross-modal alignment benefiting from the complete pipeline of Gram-Schmidt decoupling, Task Kernel Alignment, and Synergistic Representation Alignment. Meanwhile, the local task objectives and prediction interface remains intact, allowing the ablated variants to retain comparable levels of task performance. Tasks differ in their dependence on representation decoupling, peer selection, and representation alignment, which can account for the distinct degradation patterns observed in Match AUC and LinkPred AUC.

Q3: Below is the CKA matrix recorded and saved after one of the training runs: Semantic Track

TaskNodeClsLinkPredAlignRetrievalMatch
NodeCls1.00000.34160.39220.38950.4075
LinkPred0.34161.00000.59740.58230.6065
Align0.39220.59741.00000.94540.9378
Retrieval0.38950.58230.94541.00000.9133
Match0.40750.60650.93780.91331.0000
Topology Track
TaskNodeClsLinkPredAlignRetrievalMatch
NodeCls1.00000.79150.74180.70860.7869
LinkPred0.79151.00000.77710.68210.8321
Align0.74180.77711.00000.79260.8659
Retrieval0.70860.68210.79261.00000.8034
Match0.78690.83210.86590.80341.0000
Empirical records indicate that this shift occurs at two levels. For instance, in round 5, the LinkPred topology track excludes NodeCls, assigning weights of approximately 29.35%, 31.49%, and 39.15% to Align, Retrieval, and Match, respectively. By round 10—with all peers retained—the weights remain at 20.21%, 26.70%, 20.38%, and 32.71%, rather than being distributed uniformly. Consequently, the reweighting based on compatibility persists even after the curriculum concludes.
These shifts in weights alter the training signals for SRA through teacher representations and anchor confidence. When uniform guidance conflicts with the local link-prediction objective, multi-round optimization can cause this interference to accumulate, thereby weakening the model’s ability to distinguish between positive and negative edges. Thus, "selecting just one fewer peer" does not necessarily imply a negligible impact on performance; the critical factor is whether the introduced or suppressed guidance aligns with the optimization objectives of LinkPred.

Q4: Your observations are both detailed and insightful. However, there are a few points that require clarification, as some aspects may have been misleading. First, the use of different backbones leads to slight variations in robustness regarding this hyperparameter. Second, setting λ=0\lambda=0 is not strictly equivalent to removing the SRA component; due to the coupling between components, the SRA public graph representation—along with the associated GS orthogonality/reconstruction losses—is still computed even when λ=0\lambda=0. These processes inevitably influence the final results. Consequently, even if λ=0\lambda=0 is used with the same backbone and dataset, the outcomes may still differ from those reported in the ablation study. Your suggestion to expand the hyperparameter search range and ensure consistent experimental settings is highly valuable. Following the experimental setup used in Experiment 4, we conducted a small-scale evaluation with λ=0\lambda=0, λ=0.001\lambda=0.001, and λ=0.005\lambda=0.005, yielding the following results: 77.63, 78.02, and 78.46 for alignment, and 56.27, 56.98, and 57.21 for retrieval. We will include additional experiments on hyperparameters in the revised version of the paper.

Dear Reviewer, As we are nearing the end of the discussion period, we hope you have had a chance to look at our rebuttal and the revised manuscript. If they address your concerns, we would appreciate it if you could consider updating your rating; if not, please let us know what remains. Thank you for your time and feedback.

Reviewer_BZFHOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies federated multi-task learning over multimodal graphs, where different clients may pursue different downstream objectives. It focuses on two challenges: the entanglement of semantic and topological information, and the negative transfer caused by task-agnostic representation alignment. The authors propose FedM3G, which constructs semantic and topology-oriented tracks through soft Gram–Schmidt decoupling, estimates client compatibility using Task Kernel Alignment (TKA), and performs personalized alignment through Synergistic Representation Alignment (SRA). The main contribution is the combination of representation decomposition and selective collaboration for heterogeneous federated multimodal graph learning. Experiments cover multiple datasets, five downstream tasks, heterogeneous backbones, and joint domain-task heterogeneity. The reported results show consistent improvements over the considered baselines.

Critical strengths
  1. The paper considers both multimodal graph structure and task heterogeneity across clients. This reflects realistic collaborative learning scenarios in which different clients may require different semantic and topological information.
  2. Gram–Schmidt decoupling, TKA, and SRA provide a coherent pipeline for representation construction, compatibility estimation, and selective alignment. The soft gating mechanism allows the topology-oriented track to retain task-useful semantic information instead of enforcing strict orthogonality on the final representation.
  3. The authors compare against nine federated learning baselines and evaluate node classification, link prediction, cross-modal alignment, retrieval, and matching. The paper also includes joint heterogeneity experiments, module ablations, and hyperparameter sensitivity analysis.
  4. The paper is organized around two well-motivated challenges, and the three proposed modules have distinct roles. The framework figure also provides a useful overview of the client-server information flow.
Critical weaknesses
  1. The relationship between downstream tasks and the dual-track structure should be clarified.
  2. Section 4.4 introduces the public-anchor consensus, but its construction is not sufficiently specified. The paper should explain how public-anchor consensus is computed and whether the mixing coefficient is fixed, tuned, or learned.
  3. The paper should present the novelty of TKA carefully.
  4. The authors should provide a concise configuration summary in Section 5.1, including the number of clients, task allocation, and backbone architecture.
  5. Small numerical improvements should be described more cautiously.
Questions for authors

Please see the above weaknesses.

2 comments

We appreciate your recognition of the realistic collaborative setting, the coherent roles of GS, TKA, and SRA, and the clarity of the framework.

W1: The two tracks organize public-anchor knowledge transfer during training. GS constructs semantic and topology-oriented representations; TKA estimates peer compatibility separately for each track; SRA uses the resulting teachers to guide local learning. In the original prediction interface, graph-task heads consume backbone embeddings, while alignment, retrieval, and matching heads use the modality-specific features exposed by the backbone. Task dependence therefore enters through the local supervised objective, the task-head interface, and the semantic/topology alignment weights. It does not require a hard assignment of each task to exactly one track. The topology-oriented output also retains a gated semantic component: orthogonality applies to the intermediate GS residual, while the final output is a soft combination. We will clarify these training and prediction pathways in Section 4.1 and replace the ambiguous statement that each task “draws on its preferred track” with this explicit description.

W2: The public-anchor consensus is similarity-weighted: for each anchor and track, all uploading clients are weighted by agreement with the mean representation direction. The personalized peer teacher excludes the recipient. The mixing coefficient is configured, fixed within a run, and shared by both tracks; modal tasks use a configured cap when task-adaptive regularization is enabled. It is not learned by backpropagation. The complete consensus definition and mixing rule are now provided in Section 4.4, “Synergistic Representation Alignment,” including Eq. (7), of the revised manuscript (highlighted in blue).

W3: TKA builds on established centered kernel alignment (CKA). Its contribution is the use of separate semantic and topology-oriented compatibility estimates to organize personalized transfer in this federated setting. Clients evaluate the same ordered public anchors; the server compares their relational geometry by track, smooths the compatibility scores, and applies a curriculum top-kk mask and softmax weighting to construct recipient-specific teachers. This provides a compatibility signal without comparing private task labels. The score is a proxy influenced by domain, architecture, and optimization as well as task relatedness. We will distinguish the inherited CKA measure from its integration with GS and SRA in the revised paper.

W4: We will consolidate the existing configuration into a compact table in appdendix. The five-task setup uses five clients, with one client assigned to each of node classification, link prediction, alignment, retrieval, and matching. The main comparison uses MGAT, the mixed-source comparison in Table 2 uses MMGCN, and the additional backbone comparison uses MHGAT. The summary will identify the client/domain allocation for each setting and refer to the appendix for the backbone architecture and task-head details. This will make the client count, task allocation, and backbone choice directly accessible alongside the experimental protocol.

W5: We agree that the wording should reflect the magnitude and variability of the reported differences. We will describe small gains as modest numerical improvements or comparable performance, report absolute percentage-point differences where appropriate, and read them alongside the reported seed variability. A higher mean by itself does not establish statistical significance. We will revise the corresponding descriptions in the main comparison, and conclusion so that the strength of each claim matches the evidence already reported.

Thank the authors for their detailed response, which has addressed my concerns. Considering that this paper opens up a novel and important multi-task learning paradigm in the field of federated multimodal graph learning and maintains a relatively high manuscript quality, I will update my score accordingly and lean towards recommending the paper for clear acceptance.

Reviewer_1chdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates federated multi-task learning on multimodal attributed graphs and identifies two key challenges: (i) the entanglement of node-intrinsic semantics with neighbor-derived topological information during message passing, and (ii) the negative transfer induced by task-agnostic uniform alignment when private labels are inaccessible. The authors propose FedM3G, comprising Gram-Schmidt decoupling, Task Kernel Alignment (TKA), and Synergistic Representation Alignment (SRA). Extensive experiments demonstrate that FedM3G outperforms existing federated baselines across multiple datasets and five downstream tasks, with especially pronounced gains in cross-modal alignment, while ablation studies verify the effectiveness and complementarity of the three modules.

Critical strengths
  1. The problem selection and overall design are valuable.The paper situates “semantic–topological entanglement” and “task-agnostic alignment” within a unified federated multimodal graph framework and advocates a “decouple first, then align selectively according to compatibility” principle, which is closer to realistic multi-task federated scenarios than uniform representation alignment alone. The three proposed modules have clearly delineated and complementary roles, and the ablation results corroborate this division of labor.
  2. The experiments span three multimodal attributed graphs and five downstream tasks, and compare against nine federated baselines. Together with the mixed-source domain/task heterogeneity study and the ablation and sensitivity analyses, the empirical workload is substantial.
Critical weaknesses

1.Table 3 includes a “w/o GS decoupling” variant, which demonstrates that removing the entire dual-track mechanism degrades performance. However, the paper lacks ablations that retain the dual-track design but use only the semantic track or only the topology track. Such track-isolation controls are needed to substantiate the claim that the two tracks carry complementary information and that each task benefits from its preferred track. 2.Section 4.4 introduces a “robust public-anchor consensus” Cq , but Cq is not explicitly defined. The paper does not specify how Cqis computed (e.g., uniform or weighted average across clients).

Questions for authors

1.Table 3 shows that removing the entire GS decoupling module degrades all five metrics, but it does not isolate the contribution of each track. Could the authors report results for two additional variants: (i) using only the semantic track Zsem, (ii) using only the topology track Ztopo. 2.Section 4.4 introduces a “robust public-anchor consensus” Cqbut does not define how it is computed. Is Cqa uniform average of all clients’ public-anchor representations?

1 comment

We appreciate your recognition of the practical problem, the decouple-then-selectively-align design, and the breadth of our evaluation.

W1/Q1: We fully acknowledge the point you raised and have added an ablation study to address it. We will incorporate this into the main text. Specifically, the updated Table 3 includes semantic-only and topology-only prediction variants to distinguish the contribution of each track from removing the entire GS module.

The semantic-only variant achieves higher Retrieval R@5 and Match AUC than the topology-only variant, with 61.69% versus 59.40% and 79.14% versus 76.16%, respectively. The topology-only variant achieves higher LinkPred AUC, reaching 88.01% compared with 84.44% for the semantic-only variant. These differences indicate task-dependent preferences for semantic and topology-oriented information.

The Full model reports 90.90% LinkPred AUC, 63.92% Retrieval R@5, and 80.85% Match AUC. We will discuss these comparisons alongside the existing w/o GS ablation to clarify the roles of the two tracks and distinguish track-specific prediction from removing the representation construction mechanism. The added results are highlighted in blue in Table 3.

W2/Q2: Thank you for identifying the missing definition. The public-anchor consensus is similarity-weighted. For each anchor and track, all uploading clients contribute, with weights determined by agreement with the mean representation direction. The personalized peer teacher excludes the recipient. We have added the complete consensus definition to Section 4.4, “Synergistic Representation Alignment,” in Eq. (7) of the revised manuscript. The added content is highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.