- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 3(+2)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies task heterogeneity in multimodal federated learning, where the main challenge is to align cross-client knowledge while avoiding interference from misaligned local task information. The proposed method decouples alignment into semantic and topological components. The server employs a kernel-based alignment method to estimate knowledge compatibility between clients, and the resulting correlation scores are used to construct a personalized teacher for each client. While the problem setting is practical and challenging, and the proposed method is tailored to address it, several concerns remain that may affect the persuasiveness and contribution of this paper.
Critical strengths
-
The problem studied is interesting and practical. Local task structures can significantly affect the knowledge extracted from multimodal data. Moreover, the label-free setting reflects a classic compatibility challenge in typical federated learning scenarios.
-
The proposed method is well-described and well-aligned with the target problem, making the problem more tractable. Especially, the TKA design is reasonable and computationally efficient. It compares the relational geometry of representations as a label-free proxy.
-
The experimental evaluation appears comprehensive, covering 9 baselines, 3 datasets, multiple backbones, and a Q2 stress test with mixed domains and tasks. Using Fig. 1 as a preliminary experiment to motivate the proposed method is also appropriate.
Critical weaknesses
-
The metric Align CLIP only reports the alignment score of positive pairs, what about the negative pairs? Is the improvement is from the learn contrastive information or just collapsed results (especially considering the score raises up to 0.99 on some baselines) ? The improvement of the alignment score is already noticeable without the designed SRA module, which is basically how the clients share knowledge between each other, removing it means that the clients are only doing local training only. Also, the local training baseline should be included in ablation study and main comparison.
-
The ablation study shows that removing TKA leads to lower performance on some metrics than removing SRA. This suggests that incompatible knowledge sharing could perform worse than no sharing. I think further analysis is needed to better understand this phenomenon, such as examining the training curves and investigating how informative the uniformly constructed representations are for local tasks. The interactions between different modules are not sufficiently studied. It would be helpful to replace some modules with simpler designs or components adopted from existing baselines to better understand the contribution of each module compared with simpler alternatives.
-
In the ablation study, the sweep of covers the range [0.01, 0.5], a 50-fold range, yet the reported metrics barely change across this range. In contrast, Table 3 shows substantial performance changes when SRA is removed. Although the two experiments use different datasets, these results seem inconsistent w.r.t. the claim. Would a wider range of be needed to better understand its effect, or would it be more appropriate to use the same dataset for both experiments?
-
In Table 2, FedAvg and FedLPS have exact same results (even the std.) on some metrics.
-
A highly relevant benchmark work "MM-OpenFGL: A Comprehensive Benchmark for Multimodal Federated Graph Learning" is not cited or discussed in this paper.
-
The paper states that "CreamFL fails to converge on LinkPred on all three datasets because its prediction-level distillation on the public batch ..." However, to the best of my knowledge, CreamFL communicates per-sample representations of public data (a contrastive representation ensemble), not predictions or logits. Could the authors clarify this explanation?
-
A reproducibility section should be included in the appendix to facilitate the reproduction of the results for both the proposed method and the baselines.
-
Potentially incorrect references, possibly hallucinated by an LLM. The following references should be verified:
- GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner (author list and venue)
- Graph Structure Mapping Neural Networks for Generative Graph Self-Supervised Learning
Questions for authors
See the weaknesses above.
4 comments
Follow-up on the authors' response
Thank you for the response and the revision. Some of my concerns have been addressed, and I will raise my score to 2 for now. The following issues remain:
-
Incorrect references. The reference list still contains errors. Fallah et al. (2020) is titled "Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach", not "A MARS approach". The third author of Wang et al. (CVPR 2019) is Barnabás Póczos, not "Slawomir Pociot". Please check every reference against its published version.
-
Identical ablation results. In Table 3, "w/o TKA" and "w/o GS decoupling" are identical on NodeCls (43.85±0.1) and Align CLIP (83.21±0.6) and within 0.02 on Match; "w/o SRA" gives Align CLIP 83.23±0.6. Three different interventions producing the same value suggests these metrics are governed by something other than the removed module. Please confirm that these rows come from separate runs and explain why the values coincide.
-
Effect of TKA. With five clients (one per task, Table 4) and k0 = 3 (Table 6), each client has four peers, so the mask excludes at most one peer early in training and none once the curriculum ends. The only other difference from the uniform-weight ablation is the softmax reweighting. Please show the CKA matrices and the resulting peer weights over rounds, and explain how this produces an 8.64-point LinkPred drop.
-
λ sensitivity. The new sweep only extends upward (λ from 1 to 10), but the inconsistency I raised is at the lower end. Table 3 implies that removing the SRA loss (λ = 0) costs 6.9 (Align), 4.7 (Retrieval) and 7.75 (LinkPred) points, while Fig. 4 reports a 0.03% macro drift over [0.01, 0.5]. Please add λ = 0 and one or two values below 0.01 to the sweep, and run the ablation and the sweep on the same dataset and backbone.
We sincerely appreciate the reviewer’s continued engagement and insightful guidance. Your thoughtful reassessment and constructive suggestions have been invaluable in refining our discussion and improving the clarity and rigor of the manuscript.
Q1: We apologize for the inaccuracies in the references; both issues you identified are indeed correct. We will rectify these entries and conduct a thorough, item-by-item review against the original publication records. Furthermore, we will consult the original papers to verify that each citation accurately supports the corresponding claims in the text. We will also correct any imprecise descriptions of methodology and any attributions that lack sufficient evidence, incorporating these revisions into the updated manuscript.
Q2: I have reviewed the relevant experimental logs and confirmed that these results were obtained from separate runs. In fact, the raw values were not identical; for the sake of clear presentation, the figures in the paper have been rounded. Removing different components affects distinct stages of collaboration but may lead to similar losses in collaborative gains for the same downstream task. For example, all three ablated variants achieve Align CLIP scores of approximately 83.2, about 6.9 points below the full model. This pattern is consistent with cross-modal alignment benefiting from the complete pipeline of Gram-Schmidt decoupling, Task Kernel Alignment, and Synergistic Representation Alignment. Meanwhile, the local task objectives and prediction interface remains intact, allowing the ablated variants to retain comparable levels of task performance. Tasks differ in their dependence on representation decoupling, peer selection, and representation alignment, which can account for the distinct degradation patterns observed in Match AUC and LinkPred AUC.
Q3: Below is the CKA matrix recorded and saved after one of the training runs: Semantic Track
| Task | NodeCls | LinkPred | Align | Retrieval | Match |
|---|---|---|---|---|---|
| NodeCls | 1.0000 | 0.3416 | 0.3922 | 0.3895 | 0.4075 |
| LinkPred | 0.3416 | 1.0000 | 0.5974 | 0.5823 | 0.6065 |
| Align | 0.3922 | 0.5974 | 1.0000 | 0.9454 | 0.9378 |
| Retrieval | 0.3895 | 0.5823 | 0.9454 | 1.0000 | 0.9133 |
| Match | 0.4075 | 0.6065 | 0.9378 | 0.9133 | 1.0000 |
| Topology Track | |||||
| Task | NodeCls | LinkPred | Align | Retrieval | Match |
| NodeCls | 1.0000 | 0.7915 | 0.7418 | 0.7086 | 0.7869 |
| LinkPred | 0.7915 | 1.0000 | 0.7771 | 0.6821 | 0.8321 |
| Align | 0.7418 | 0.7771 | 1.0000 | 0.7926 | 0.8659 |
| Retrieval | 0.7086 | 0.6821 | 0.7926 | 1.0000 | 0.8034 |
| Match | 0.7869 | 0.8321 | 0.8659 | 0.8034 | 1.0000 |
| Empirical records indicate that this shift occurs at two levels. For instance, in round 5, the LinkPred topology track excludes NodeCls, assigning weights of approximately 29.35%, 31.49%, and 39.15% to Align, Retrieval, and Match, respectively. By round 10—with all peers retained—the weights remain at 20.21%, 26.70%, 20.38%, and 32.71%, rather than being distributed uniformly. Consequently, the reweighting based on compatibility persists even after the curriculum concludes. | |||||
| These shifts in weights alter the training signals for SRA through teacher representations and anchor confidence. When uniform guidance conflicts with the local link-prediction objective, multi-round optimization can cause this interference to accumulate, thereby weakening the model’s ability to distinguish between positive and negative edges. Thus, "selecting just one fewer peer" does not necessarily imply a negligible impact on performance; the critical factor is whether the introduced or suppressed guidance aligns with the optimization objectives of LinkPred. |
Q4: Your observations are both detailed and insightful. However, there are a few points that require clarification, as some aspects may have been misleading. First, the use of different backbones leads to slight variations in robustness regarding this hyperparameter. Second, setting is not strictly equivalent to removing the SRA component; due to the coupling between components, the SRA public graph representation—along with the associated GS orthogonality/reconstruction losses—is still computed even when . These processes inevitably influence the final results. Consequently, even if is used with the same backbone and dataset, the outcomes may still differ from those reported in the ablation study. Your suggestion to expand the hyperparameter search range and ensure consistent experimental settings is highly valuable. Following the experimental setup used in Experiment 4, we conducted a small-scale evaluation with , , and , yielding the following results: 77.63, 78.02, and 78.46 for alignment, and 56.27, 56.98, and 57.21 for retrieval. We will include additional experiments on hyperparameters in the revised version of the paper.
Dear Reviewer, As we are nearing the end of the discussion period, we hope you have had a chance to look at our rebuttal and the revised manuscript. If they address your concerns, we would appreciate it if you could consider updating your rating; if not, please let us know what remains. Thank you for your time and feedback.
We appreciate your recognition of the practical problem, the label-free compatibility design, and the breadth of our experimental evaluation.
W1: We appreciate your question. In the experiments presented in the main table, the cosine similarity between positive pairs is significantly higher than that between negative pairs. In this implementation, disabling SRA removes the representation-alignment loss while maintaining the communication loop and the server-provided prototype terms. The "Local-only" setting further disables communication and public-data updates, while also clearing all server-provided guidance and prototypes. Since there is only one client per task, task-grouped prototypes may originate from that same client; therefore, the mere retention of the communication loop does not constitute cross-client transfer. If all server-dependent updates are disabled, the training process effectively reduces to the "Local-only" setting. We will include Local train as a baseline in the main text.
W2: Removing TKA retains cross-client alignment but removes compatibility-based peer selection. Incompatible guidance can therefore compete with the local task objective, whereas removing SRA removes that alignment pressure. This provides a mechanism through which unselective sharing can perform worse than disabling alignment.
The semantic-only and topology-only rows in Table 3 show different preferences, with semantic features favoring retrieval and matching and topology-oriented features favoring link prediction. These comparisons motivate separating the tracks before selecting transfer targets. They assess track contribution, while peer-selection effects are addressed by the uniform-weight controls.
We also tested uniform peer weights and replaced GS gating/projection with direct self/aggregate features. In the strict comparison, retaining GS improves Retrieval under both CKA and uniform peer weights.
W3: Thank you very much for your suggestion. We examined the wider range of settings you proposed. As λ increases from 1 to 10, the MR drops from 57.21% to 33.86%; this change becomes even more pronounced at higher weight values. This pattern suggests that an excessively strong alignment constraint may interfere with the objectives of local tasks. However, the range where such significant fluctuations occur falls outside the scope of typical hyperparameter selection.
W4: Thank you for highlighting the identical FedAvg/FedLPS results. We rerun the experiment, verified the execution logs and reproduces the reported means and standard deviations. A plausible explanation is that the two methods reduce to the same effective local prediction process for these tasks under the adopted backbone and task-head interface. FedAvg and FedLPS differ in how shared parameters are updated and constrained. Identical results arise when the differences between methods do not affect the effective prediction pathway of a task. Our controlled reruns support this explanation. Before and after activating FedLPS freezing and pruning changes graph-task performance while leaving the three modal tasks unchanged.
W5: Thank you for pointing out this relevant benchmark. We will discuss MM-OpenFGL as the broader benchmark context.
W6: We would like to clarify the communication object and the scope of our comparison. Our implementation uploads per-sample public representations and trains the server against aggregated representation targets using a representation-level MSE loss. This is consistent with the representation-based communication described in CreamFL). We will replace “prediction-level distillation” with “representation-level distillation” to accurately describe the implemented operation.
The relevant distinction for LinkPred concerns the learning objective. Matching representations on public nodes does not directly optimize the scores of positive and negative edges on a client's private graph. Edge discrimination also depends on the graph encoder, edge-scoring head, local supervision, and interaction with the shared objective. This distinction explains why public representation matching alone does not guarantee improved LinkPred performance. It does not, by itself, establish the cause of the observed low AUC.
W7: We appreciate your recommendations for clearer configuration and reproducibility reporting. We will add a table in appendix covering more detailed information.
W8: To be honest, we did not actually use these two backbones in our experiments; they were simply included during our literature review and data collection process. Removing these two references will not affect the readability of the paper. Regarding the incorrect references, we fully acknowledge the mistak that we copied the wrong citation sources and failed to provide the references in the correct manner.