- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies federated adaptation of multimodal graph models that combine a multimodal Encoder with a GNN. Three diagnostic studies are used to argue that Encoder adaptation is necessary, that it depends on graph relations, and that separately parameterized Encoder and GNN updates do not combine well across clients. The proposed FedCORE gives each client one r × r core, decodes it through fixed SVD coordinates of the pretrained matrices into low-rank updates for the Encoder, fusion and GNN layers, and has clients upload only the core and a coordination score. On eight datasets covering five tasks FedCORE has the best mean result in every column of Table 1, and on Toys the paper reports a pairing gap of 5.9 against 30.6 for Independent Joint LoRA.
Critical strengths
-
The paper raises a useful question for federated multimodal graph learning, namely whether Encoder and GNN adaptations stay compatible once they are trained on different clients and aggregated. The cross-client swapping test in Section 3.3 turns this question into something measurable, which other work on federated adaptation can reuse. This framing is the main point in favour of acceptance, even though the present evidence for it is incomplete.
-
Decoding every module update from one core through fixed linear maps makes federated aggregation exact by construction. Averaging the cores gives the same result as averaging the decoded module updates (Proposition A.3), so the aggregated model stays in the same family of updates that clients train, and the communicated state does not grow with the number of adapted modules. The appendix is also candid about what this construction does not guarantee, which makes the method easy to check and to build on.
Critical weaknesses
-
The pairing study in Figure 3 shows pairing sensitivity but does not isolate why it occurs or which part of the design reduces it. Appendix A.3.3 gives a common evaluation protocol but does not identify the evaluation data or report a control that moves both modules together to another client, so client mismatch may contribute to the gap. For FedCORE, similar client updates could also reduce the gap, and Figure 5 shows a shared-core similarity of roughly 89% to 95% between FedCORE clients on Toys, while both methods report the same matched-pair accuracy of 92.4%. The appendix itself concedes that the diagnostic does not isolate the effect of core sharing from coordinate anchoring, and the study covers one dataset and five clients with no variation across runs reported. Observation 3 and the only quantitative result in the abstract both rest on this figure, so this weakness is the main reason for my recommendation.
-
The accuracy results do not show how much of FedCORE's margin over the baselines comes from the shared core. In Figure 6(b) the r = 1 row, where all modules share a single scalar, reaches an AUC of 78.3 to 79.5 on Bili Music, close to the r = 16 row (78.6 to 79.8) and above every baseline in Table 1. Table 2 shows losses of 1.5 to 2.9 points when sharing is removed, but no result reports FedCORE after the warm-up stage with the core fixed at zero, so what the core adds beyond the warm-up stage and the client-specific task heads remains unmeasured. The Conclusion nevertheless presents the gains as evidence for the importance of coordinated perception and reasoning adaptation.
Questions for authors
-
Which test sets and how many runs produce the three panels of Figure 3, which Appendix A.3.3 says share one evaluation protocol? All 50 entries in the two baseline panels match a fraction with denominator 140 or 141 once rounded to one decimal, as a single run on about 140 test nodes would give, whereas most entries in the FedCORE panel do not. Please also state which task head a swapped pair uses.
-
What does FedCORE score on the Table 1 datasets when the core is fixed at zero after the warm-up stage? The README of the supplied code says that the base weights are learned in a federated warm-up and that each client trains its own task head, so this number is needed to see what the core adds.
-
Why are Frozen Enc-GNN and Joint LoRA level with FedCORE at about 85% at zero corruption in all four panels of Figure 7, but about four points below it on Grocery in Table 1?
-
Does the adapted multimodal Encoder consist only of the visual, textual and fusion projections over precomputed CLIP ViT-L/14 features described in Appendix A.5.3 and the supplied code, or does FedCORE also adapt layers inside CLIP?
-
LoRA-XS (Bałazy et al., ECAI 2025) also trains an r × r matrix between frozen SVD bases of the pretrained weights. Apart from sharing one core across modules, how do the decoder choices in Eq. (2) compare with those of LoRA-XS?
References for the works named above
Klaudia Bałazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters. ECAI 2025, Frontiers in Artificial Intelligence and Applications, Volume 413. DOI 10.3233/FAIA251185.
3 comments
Response to Reviewer wbnN (Q3, Q4, Q5)
Q3 — Inconsistency Between Figure 7 and Table 1 Under Zero Corruption
We thank the reviewer for identifying the inconsistency between Figure 7 and Table 1 under zero corruption. Our experimental records suggest that the original robustness figure may have used a separately selected checkpoint rather than the main experimental pipeline, resulting in insufficient alignment of baseline configurations and reporting protocols. To address this issue, we reran the Grocery robustness experiments using the main experimental pipeline's parameter settings, covering image corruption, text corruption, missing images, and missing text, and recomputed the zero-corruption accuracies for all evaluated methods under consistent settings.
The corrected robustness results are presented in Figure 7 of Sec. 5.4, while the main performance results and corresponding experimental configurations are documented in Table 1 and Appendix A.5, respectively.
Q4 — Are the Internal Layers of CLIP Fine-Tuned?
We thank the reviewer for requesting clarification of the Encoder adaptation scope. In our experiments, CLIP ViT-L/14 serves as a frozen feature extractor, and its internal layers are not fine-tuned by FedCORE. Instead, perception-side adaptation is applied to the visual projection, textual projection, and fusion layers built on top of the precomputed CLIP features, which are jointly adapted with the GNN transformation matrices through the shared latent core.
We have clarified this distinction in Appendix A.5.4 ("Multimodal backbone and CLIP usage") to explicitly distinguish projection/fusion-layer adaptation from CLIP backbone fine-tuning.
Q5 — Relationship Between FedCORE and LoRA-XS
We thank the reviewer for highlighting the relationship between FedCORE and LoRA-XS. Both approaches learn an core matrix between fixed truncated SVD bases. Under a unified notation, LoRA-XS can be expressed as
where singular-value scaling is absorbed into one fixed factor. In contrast, FedCORE adopts
where contains the fourth roots of normalized singular values, providing bilateral spectral scaling, while compensates for scale differences across heterogeneous modules. For an individual module, when the scaling factors are invertible, they can be absorbed into the core, meaning that both parameterizations span the same truncated-SVD update subspace. The fundamental distinction is that LoRA-XS learns an independent core for each weight matrix, whereas FedCORE shares a common core across perception and reasoning modules. This shared state imposes joint constraints across heterogeneous module updates and, under fixed linear decoders, preserves their coupled parameterization during federated aggregation. Thus, FedCORE's contribution lies not in introducing another single-matrix SVD adaptation subspace, but in establishing cross-module coupling and aggregation closure in federated adaptation.
We have added the LoRA-XS reference in Sec. 2.2 and clarified these distinctions in Appendix A.4 ("Relation to LoRA-XS").
Thank you for the response
Thank you for the detailed responses and new experiments, and although some claims about the shared core are not yet fully resolved, I find this an interesting and meaningful work and have raised my rating to 3.
Response to Reviewer wbnN (W1, W2, Q1, Q2)
W1 & Q1 — Confounding Factors in the Pairing Gap and Evidence for the Shared-Core Mechanism
We thank the reviewer for carefully examining the pairing experiment and the precision of the reported results. Our experimental records indicate that the two baseline panels were originally obtained from separate single-run evaluation pipelines, whereas the FedCORE panel was summarized over seven random seeds (42–48). We have now extended both baseline evaluations to the same seven seeds and supplemented the results with variability statistics. For each pairing (Ei,Gj), we fix client j's GNN, task head Hj, graph structure, and complete test split Dj, replacing only the Encoder with Ei. Thus, all entries within the same column share identical evaluation conditions, with diagonal entries representing within-client pairings and off-diagonal entries representing cross-client recombinations.
Independent Joint LoRA achieves a matched-pair accuracy of 78.4%, which drops to 52.8% under cross-client mismatching, yielding a reported pairing gap of 25.5 percentage points. Module-wise FedAvg reduces this gap to 11.4 percentage points but remains sensitive to module swapping, whereas FedCORE achieves a mismatched-pair accuracy of 78.1% and further reduces the gap to 6.4 percentage points. To disentangle fixed coordinate anchoring from core sharing, we additionally compare FedCORE against a parameter-budget-matched Independent Core baseline that retains the same fixed SVD bases. Independent Core uses two separate 11×11 cores (242 trainable parameters), whereas FedCORE employs a shared 16×16 core (256 parameters).
Independent Core achieves matched/mismatched accuracies of 81.0%/72.9%, corresponding to an 8.1-point pairing gap, while FedCORE achieves 84.5%/78.1%, reducing the gap to 6.4 points. FedCORE therefore improves mismatched accuracy by 5.2 percentage points and reduces the pairing gap by 1.7 percentage points. These results substantiate Observation 3: independently adapted Encoder–GNN updates may exhibit reduced compatibility under cross-client recombination, whereas a shared latent core helps mitigate module-swapping sensitivity. The theoretical analysis provides a complementary structural explanation: under fixed linear decoders, averaging shared cores preserves the common latent-state constraint across module updates. Thus, pairing experiments quantify practical recombination performance, while aggregation closure establishes the preservation of coupled parameterization during federated aggregation.
The revised Figure 3 reports means and standard deviations over seven seeds; Appendix A.3.4 specifies the evaluation protocol, Figure 9 presents the Independent Core comparison, and Propositions A.2–A.3 in Appendix A.4 establish the shared parameterization and aggregation-closure property.
W2 & Q2 — What Does the Shared Core Contribute Beyond Warm-up and Local Task Heads?
We appreciate this question and introduce a Warm-up + Zero-Core baseline to directly isolate the contribution of trainable core adaptation. This baseline retains federated warm-up and client-specific task heads but fixes the adaptation core to zero after warm-up. We compare it against FedCORE with r=1 and r=16 across four benchmarks:
Compared with Zero-Core, FedCORE (r=16) achieves improvements of 5.08, 5.06, 2.53, and 10.67 percentage points, respectively, demonstrating that learned core adaptation provides substantial additional performance gains beyond federated warm-up and local task heads. Together with the shared-state ablation in Table 2, these results strengthen the empirical evidence for trainable core adaptation. We also plan to extend the Zero-Core comparison to the remaining benchmarks.
The new results are included in Appendix A.6.3 and Table 6, with the corresponding discussion in Sec. 5.3.