acceptodds
Under review as a conference paper at ICLR 2027

Coupling Perception and Reasoning in Federated Multimodal Graph Foundation Models

Abstract

Federated multimodal graph foundation models (GFMs) aim to adapt pretrained multimodal models to decentralized graph data, where each client owns a private multimodal graph and cannot share raw information. These models typically combine a multimodal Encoder that extracts semantic evidence from heterogeneous modalities and a graph neural network (GNN) that performs relational reasoning over graph structures. However, existing federated GFM adaptation methods mainly update graph-side modules while keeping the multimodal Encoder frozen, limiting adaptation to how information is propagated while fixing what information is extracted. Through empirical studies, we reveal that Encoder and GNN adaptations are not independent: Encoder adaptation is affected by graph relations, while cross-client module swapping reveals substantial pairing sensitivity between separately parameterized Encoder and GNN updates. Motivated by this observation, we propose FedCORE, a federated adaptation framework that represents Encoder and GNN updates through a shared low-dimensional latent state. FedCORE jointly optimizes this core from multimodal and structural signals and performs federated evolution directly in the shared state space, preserving compatibility between perception and reasoning adaptations. Extensive experiments demonstrate that FedCORE reduces the Encoder–GNN pairing gap from to , corresponding to an reduction over independent joint adaptation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Federated Graph Learning, Multimodal Graph Learning, Graph Foundation Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wbnNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies federated adaptation of multimodal graph models that combine a multimodal Encoder with a GNN. Three diagnostic studies are used to argue that Encoder adaptation is necessary, that it depends on graph relations, and that separately parameterized Encoder and GNN updates do not combine well across clients. The proposed FedCORE gives each client one r × r core, decodes it through fixed SVD coordinates of the pretrained matrices into low-rank updates for the Encoder, fusion and GNN layers, and has clients upload only the core and a coordination score. On eight datasets covering five tasks FedCORE has the best mean result in every column of Table 1, and on Toys the paper reports a pairing gap of 5.9 against 30.6 for Independent Joint LoRA.

Critical strengths
  1. The paper raises a useful question for federated multimodal graph learning, namely whether Encoder and GNN adaptations stay compatible once they are trained on different clients and aggregated. The cross-client swapping test in Section 3.3 turns this question into something measurable, which other work on federated adaptation can reuse. This framing is the main point in favour of acceptance, even though the present evidence for it is incomplete.

  2. Decoding every module update from one core through fixed linear maps makes federated aggregation exact by construction. Averaging the cores gives the same result as averaging the decoded module updates (Proposition A.3), so the aggregated model stays in the same family of updates that clients train, and the communicated state does not grow with the number of adapted modules. The appendix is also candid about what this construction does not guarantee, which makes the method easy to check and to build on.

Critical weaknesses
  1. The pairing study in Figure 3 shows pairing sensitivity but does not isolate why it occurs or which part of the design reduces it. Appendix A.3.3 gives a common evaluation protocol but does not identify the evaluation data or report a control that moves both modules together to another client, so client mismatch may contribute to the gap. For FedCORE, similar client updates could also reduce the gap, and Figure 5 shows a shared-core similarity of roughly 89% to 95% between FedCORE clients on Toys, while both methods report the same matched-pair accuracy of 92.4%. The appendix itself concedes that the diagnostic does not isolate the effect of core sharing from coordinate anchoring, and the study covers one dataset and five clients with no variation across runs reported. Observation 3 and the only quantitative result in the abstract both rest on this figure, so this weakness is the main reason for my recommendation.

  2. The accuracy results do not show how much of FedCORE's margin over the baselines comes from the shared core. In Figure 6(b) the r = 1 row, where all modules share a single scalar, reaches an AUC of 78.3 to 79.5 on Bili Music, close to the r = 16 row (78.6 to 79.8) and above every baseline in Table 1. Table 2 shows losses of 1.5 to 2.9 points when sharing is removed, but no result reports FedCORE after the warm-up stage with the core fixed at zero, so what the core adds beyond the warm-up stage and the client-specific task heads remains unmeasured. The Conclusion nevertheless presents the gains as evidence for the importance of coordinated perception and reasoning adaptation.

Questions for authors
  1. Which test sets and how many runs produce the three panels of Figure 3, which Appendix A.3.3 says share one evaluation protocol? All 50 entries in the two baseline panels match a fraction with denominator 140 or 141 once rounded to one decimal, as a single run on about 140 test nodes would give, whereas most entries in the FedCORE panel do not. Please also state which task head a swapped pair uses.

  2. What does FedCORE score on the Table 1 datasets when the core is fixed at zero after the warm-up stage? The README of the supplied code says that the base weights are learned in a federated warm-up and that each client trains its own task head, so this number is needed to see what the core adds.

  3. Why are Frozen Enc-GNN and Joint LoRA level with FedCORE at about 85% at zero corruption in all four panels of Figure 7, but about four points below it on Grocery in Table 1?

  4. Does the adapted multimodal Encoder consist only of the visual, textual and fusion projections over precomputed CLIP ViT-L/14 features described in Appendix A.5.3 and the supplied code, or does FedCORE also adapt layers inside CLIP?

  5. LoRA-XS (Bałazy et al., ECAI 2025) also trains an r × r matrix between frozen SVD bases of the pretrained weights. Apart from sharing one core across modules, how do the decoder choices in Eq. (2) compare with those of LoRA-XS?

References for the works named above

Klaudia Bałazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters. ECAI 2025, Frontiers in Artificial Intelligence and Applications, Volume 413. DOI 10.3233/FAIA251185.

3 comments
AuthorsOpenReview ↗

Response to Reviewer wbnN (W1, W2, Q1, Q2)

W1 & Q1 — Confounding Factors in the Pairing Gap and Evidence for the Shared-Core Mechanism

We thank the reviewer for carefully examining the pairing experiment and the precision of the reported results. Our experimental records indicate that the two baseline panels were originally obtained from separate single-run evaluation pipelines, whereas the FedCORE panel was summarized over seven random seeds (42–48). We have now extended both baseline evaluations to the same seven seeds and supplemented the results with variability statistics. For each pairing (Ei,Gj)(E_i,G_j), we fix client jj's GNN, task head HjH_j, graph structure, and complete test split DjD_j, replacing only the Encoder with EiE_i. Thus, all entries within the same column share identical evaluation conditions, with diagonal entries representing within-client pairings and off-diagonal entries representing cross-client recombinations.

Independent Joint LoRA achieves a matched-pair accuracy of 78.4%, which drops to 52.8% under cross-client mismatching, yielding a reported pairing gap of 25.5 percentage points. Module-wise FedAvg reduces this gap to 11.4 percentage points but remains sensitive to module swapping, whereas FedCORE achieves a mismatched-pair accuracy of 78.1% and further reduces the gap to 6.4 percentage points. To disentangle fixed coordinate anchoring from core sharing, we additionally compare FedCORE against a parameter-budget-matched Independent Core baseline that retains the same fixed SVD bases. Independent Core uses two separate 11×1111\times11 cores (242 trainable parameters), whereas FedCORE employs a shared 16×1616\times16 core (256 parameters).

Independent Core achieves matched/mismatched accuracies of 81.0%/72.9%, corresponding to an 8.1-point pairing gap, while FedCORE achieves 84.5%/78.1%, reducing the gap to 6.4 points. FedCORE therefore improves mismatched accuracy by 5.2 percentage points and reduces the pairing gap by 1.7 percentage points. These results substantiate Observation 3: independently adapted Encoder–GNN updates may exhibit reduced compatibility under cross-client recombination, whereas a shared latent core helps mitigate module-swapping sensitivity. The theoretical analysis provides a complementary structural explanation: under fixed linear decoders, averaging shared cores preserves the common latent-state constraint across module updates. Thus, pairing experiments quantify practical recombination performance, while aggregation closure establishes the preservation of coupled parameterization during federated aggregation.

The revised Figure 3 reports means and standard deviations over seven seeds; Appendix A.3.4 specifies the evaluation protocol, Figure 9 presents the Independent Core comparison, and Propositions A.2–A.3 in Appendix A.4 establish the shared parameterization and aggregation-closure property.

W2 & Q2 — What Does the Shared Core Contribute Beyond Warm-up and Local Task Heads?

We appreciate this question and introduce a Warm-up + Zero-Core baseline to directly isolate the contribution of trainable core adaptation. This baseline retains federated warm-up and client-specific task heads but fixes the adaptation core to zero after warm-up. We compare it against FedCORE with r=1r=1 and r=16r=16 across four benchmarks:

DatasetWarm-up + Zero-CoreFedCORE (r=1r=1)FedCORE (r=16r=16)
Toys (Acc)76.64 ± 0.3678.38 ± 0.3481.72 ± 0.10
Grocery (Acc)79.62 ± 0.1381.77 ± 0.0784.68 ± 0.18
Bili Music (AUC)76.97 ± 0.8777.62 ± 0.7379.50 ± 0.94
QB (R@5)84.38 ± 0.3087.51 ± 0.2595.05 ± 1.61

Compared with Zero-Core, FedCORE (r=16r=16) achieves improvements of 5.08, 5.06, 2.53, and 10.67 percentage points, respectively, demonstrating that learned core adaptation provides substantial additional performance gains beyond federated warm-up and local task heads. Together with the shared-state ablation in Table 2, these results strengthen the empirical evidence for trainable core adaptation. We also plan to extend the Zero-Core comparison to the remaining benchmarks.

The new results are included in Appendix A.6.3 and Table 6, with the corresponding discussion in Sec. 5.3.

AuthorsOpenReview ↗

Response to Reviewer wbnN (Q3, Q4, Q5)

Q3 — Inconsistency Between Figure 7 and Table 1 Under Zero Corruption

We thank the reviewer for identifying the inconsistency between Figure 7 and Table 1 under zero corruption. Our experimental records suggest that the original robustness figure may have used a separately selected checkpoint rather than the main experimental pipeline, resulting in insufficient alignment of baseline configurations and reporting protocols. To address this issue, we reran the Grocery robustness experiments using the main experimental pipeline's parameter settings, covering image corruption, text corruption, missing images, and missing text, and recomputed the zero-corruption accuracies for all evaluated methods under consistent settings.

The corrected robustness results are presented in Figure 7 of Sec. 5.4, while the main performance results and corresponding experimental configurations are documented in Table 1 and Appendix A.5, respectively.

Q4 — Are the Internal Layers of CLIP Fine-Tuned?

We thank the reviewer for requesting clarification of the Encoder adaptation scope. In our experiments, CLIP ViT-L/14 serves as a frozen feature extractor, and its internal layers are not fine-tuned by FedCORE. Instead, perception-side adaptation is applied to the visual projection, textual projection, and fusion layers built on top of the precomputed CLIP features, which are jointly adapted with the GNN transformation matrices through the shared latent core.

We have clarified this distinction in Appendix A.5.4 ("Multimodal backbone and CLIP usage") to explicitly distinguish projection/fusion-layer adaptation from CLIP backbone fine-tuning.

Q5 — Relationship Between FedCORE and LoRA-XS

We thank the reviewer for highlighting the relationship between FedCORE and LoRA-XS. Both approaches learn an r×rr\times r core matrix between fixed truncated SVD bases. Under a unified notation, LoRA-XS can be expressed as

ΔWj=UjΣjRjVj⊤,\Delta W_j=U_j\Sigma_jR_jV_j^\top,

where singular-value scaling is absorbed into one fixed factor. In contrast, FedCORE adopts

ΔWk,j=αjUjSjCkSjVj⊤,\Delta W_{k,j}=\alpha_jU_jS_jC_kS_jV_j^\top,

where SjS_j contains the fourth roots of normalized singular values, providing bilateral spectral scaling, while αj\alpha_j compensates for scale differences across heterogeneous modules. For an individual module, when the scaling factors are invertible, they can be absorbed into the core, meaning that both parameterizations span the same truncated-SVD update subspace. The fundamental distinction is that LoRA-XS learns an independent core RjR_j for each weight matrix, whereas FedCORE shares a common core CkC_k across perception and reasoning modules. This shared state imposes joint constraints across heterogeneous module updates and, under fixed linear decoders, preserves their coupled parameterization during federated aggregation. Thus, FedCORE's contribution lies not in introducing another single-matrix SVD adaptation subspace, but in establishing cross-module coupling and aggregation closure in federated adaptation.

We have added the LoRA-XS reference in Sec. 2.2 and clarified these distinctions in Appendix A.4 ("Relation to LoRA-XS").

Reviewer_wbnNOpenReview ↗

Thank you for the response

Thank you for the detailed responses and new experiments, and although some claims about the shared core are not yet fully resolved, I find this an interesting and meaningful work and have raised my rating to 3.

Reviewer_saSHOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies federated adaptation of multimodal graph foundation models, focusing on the interaction between multimodal encoders and GNNs. The authors show that adapting the GNN alone cannot fully compensate for a frozen encoder, encoder adaptation depends on graph structure, and independently adapted encoder–GNN pairs may not work well across clients.

To address these issues, the authors propose FedCORE, which uses a shared low-dimensional latent space to coordinate updates to the encoder, fusion module, and GNN. Fixed SVD-based bases are used to derive module-specific updates from the shared representation. During federated training, clients share only the latent representation for aggregation.

Experiments on eight benchmarks covering classification, link prediction, retrieval, and generation show that FedCORE outperforms the strongest baseline by an average of 2.27%. It also reduces communication from 4.64 × 10^5 to 5.12 × 10^2 scalars per round compared with Independent Joint LoRA.

Critical strengths

1. The motivation for the method is clear.

The authors present three observations to motivate FedCORE: GNN adaptation alone cannot replace encoder adaptation, encoder updates are affected by graph structure, and independently adapted Encoder–GNN pairs may not work well across clients. Figure 3 provides a clear example of the last issue. The appendix also explains why the shared latent space helps keep the module updates consistent during aggregation.

2. The method performs well across tasks and reduces communication costs.

FedCORE achieves the best results in the main experiments across different tasks. The paper also includes ablation studies and tests under different client settings, modality corruption, and hyperparameter choices. In Figure 3, FedCORE reduces the Encoder–GNN pairing gap from 30.6 to 5.9. It also requires only 512 scalars per round, compared with hundreds of thousands for independent joint LoRA. This makes the method both effective and communication-efficient.

Critical weaknesses

1. The role of the synergy loss is unclear.

Section 4.3 introduces a synergy loss to encourage joint adaptation of perception and reasoning. However, since the isolated-branch losses in Eq. (6) are detached, the gradient of this term appears to follow that of the joint task loss when the hinge is active. In this case, the synergy loss seems to mainly change the weight of the joint loss rather than provide an additional signal for coordination. The paper does not include an ablation without this term, so it is difficult to tell whether it actually improves the results.

2. It is hard to tell what drives the performance gains.

FedCORE uses only 256 trainable parameters, compared with about 230,000 for Independent Joint LoRA, according to Table 3. This large gap raises the question of whether the improvements come from coupling perception and reasoning or simply from using a much smaller representation. A comparison with independent low-dimensional updates under a similar parameter budget would help clarify this.

The results in Figure 3 also leave some room for interpretation. Matched Encoder–GNN pairs may perform better because both modules are trained on the same client's data, not necessarily because they share a latent state. Since Table 2 already includes variants without the shared latent state and with independent adapters, reporting their pairing gaps under the same setting as Figure 3 would help clarify how much the shared core contributes.

Questions for authors

1. Does the synergy loss bring a clear benefit?

Since the isolated-branch losses are detached, the synergy loss seems to mainly reweight the joint-task gradient when the hinge is active. Could the authors clarify its effect on training? An ablation without this term would help show whether it makes a meaningful difference.

2. Could the authors clarify the parameter counts and compare against a smaller independent baseline?

Table 3 reports 256 trainable parameters for FedCORE and about 230,000 for Independent Joint LoRA. Do these counts cover the same components, including the local task heads? It would also be useful to compare FedCORE with independent low-dimensional updates under a similar parameter budget, to see whether sharing the latent state is the main source of the gains.

3. Could the authors further examine the pairing results?

Figure 3 shows that matched Encoder–GNN pairs work better than mismatched pairs, but this may partly reflect adaptation to the same client's data. Could the authors report the pairing gaps for the variants in Table 2 under the same evaluation setting? This would help clarify whether the shared core itself improves cross-client compatibility.

2 comments
AuthorsOpenReview ↗

Response to Reviewer saSH

W1 & Q1 — What Does Synergy Loss Actually Contribute?

We thank the reviewer for analyzing the gradient behavior of Synergy Loss. When the hinge term is active, the stop-gradient treatment of the reference branches makes this term conditionally reweight the joint task gradient without altering its direction.

We evaluate its practical contribution through two controlled comparisons: a learning-rate-tuned Task-only baseline and a variant without Synergy Loss. Full FedCORE achieves the highest reported mean performance across all five tasks, demonstrating the benefit of conditional reweighting under the evaluated settings.

MethodToys (Acc)DY (AUC)QB (R@5)Flickr30k (ROUGE-L)SemArt (CLIP-S)
Full FedCORE81.67 ± 0.0577.72 ± 0.3295.28 ± 0.6133.21 ± 0.7673.26 ± 0.21
Task-only (LR-tuned)81.29 ± 0.9876.54 ± 0.3894.78 ± 1.1732.32 ± 0.4772.97 ± 0.67
w/o Synergy Loss80.44 ± 0.2576.28 ± 0.1293.58 ± 0.8731.98 ± 0.4671.98 ± 0.57

Synergy Loss is an auxiliary optimization component rather than the central methodological contribution of FedCORE.

Our key insight is that graph relationships influence perception adaptation, while independently adapted perception and reasoning modules may become incompatible when recombined across clients.

FedCORE addresses this challenge by representing and generating both types of updates through a shared latent state. With fixed linear decoders, aggregating this state preserves the coupled parameterization of the resulting module updates.

This coupling across heterogeneous modules and federated aggregation constitutes the central contribution of our framework.

The optimization interpretation is provided in Sec. 4.3 and Appendix A.6.1. The comparisons against learning-rate-tuned Task-only and w/o Synergy Loss are reported in Table 5.

W2 & Q2 — Parameter Accounting and Comparison with a Small Independent Model

We appreciate the reviewer's request for clarification regarding parameter accounting and baseline configurations.

The parameter counts reported in Table 3 include only trainable adaptation parameters and exclude client-specific task heads. Importantly, the "w/o Shared Latent State" ablation in Table 2 already provides an approximately parameter-budget-matched independent-core comparison. Specifically, the perception and reasoning modules each employ an independent 11×1111\times11 core, yielding 2×112=2422\times11^2=242 trainable parameters. Full FedCORE instead uses a single shared 16×1616\times16 core with 256 trainable parameters. Both variants retain the fixed SVD bases of their respective modules. The essential difference is that Independent Core removes core sharing between perception and reasoning updates. For clarity, we consistently refer to this variant as Independent Core.

The results are summarized below.

MethodAdapt. ParamsToys (Acc)DY (AUC)QB (R@5)Flickr30k (ROUGE-L)SemArt (CLIP-S)
Independent Core24279.96 ± 0.2174.86 ± 0.4392.94 ± 0.4630.71 ± 0.9471.75 ± 0.36
FedCORE25681.67 ± 0.0577.72 ± 0.3295.28 ± 0.6133.21 ± 0.7673.26 ± 0.21

With approximately matched adaptation parameter budgets, FedCORE consistently outperforms Independent Core across all five tasks. These results indicate that the performance gains associated with the shared latent state cannot be explained solely by a smaller adaptation parameter budget.

We have clarified the parameter-budget-matched Independent Core baseline in Sec. 5.3 and Table 2. Appendix A.5.3 further specifies that the adaptation parameter counts and communication costs reported in Table 4 exclude client-specific task heads.

W3 & Q3 — Interpretation and Evaluation of the Pairing Gap

We thank the reviewer for suggesting a more direct examination of the pairing gap.

We have conducted an additional cross-client pairing analysis using Independent Core, corresponding to the "w/o Shared Latent State" variant in Table 2. Independent Core employs two separate 11×1111\times11 cores, providing an adaptation parameter budget approximately matched to FedCORE's shared 16×1616\times16 core. Both methods follow the same module-swapping evaluation protocol, with identical test splits and task-head assignment rules. (Fig.9 in our revision version)

FedCORE improves mismatched-pair accuracy by 5.2 percentage points and reduces the pairing gap by 1.7 percentage points relative to Independent Core. This provides additional empirical evidence that sharing the latent core helps reduce sensitivity to cross-client module recombination.

We have added the pairing matrices and pairing-gap comparisons for Independent Core and FedCORE in Appendix A.3.4 and Figure 9, with a corresponding discussion in Sec. 5.3.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns. I will maintain my rating of 3.

Reviewer_xkiFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies coordinated adaptation of multimodal encoders and GNNs in federated multimodal graph models. Motivated by relationally conditioned encoder updates and cross-client pairing sensitivity, FedCORE uses a shared low-dimensional core to generate module-specific updates through fixed SVD-based decoders. Local training combines task supervision, a detached-reference synergy term, and matrix preconditioning, while the server aggregates cores using coordination- and drift-dependent weights. FedCORE outperforms baselines across five task families, reduced pairing sensitivity, and substantially lower adaptation-stage communication than independent Joint LoRA.

Critical strengths
  1. Jointly considering multimodal perception and graph reasoning addresses an important challenge in coordinating module adaptation across heterogeneous clients and provides a useful direction for federated multimodal graph learning.

  2. A shared low-dimensional core controls updates across heterogeneous modules, explicitly coupling their adaptation while avoiding the transmission of numerous independent adapter parameters.

  3. FedCORE achieves competitive results on node classification, link prediction, cross-modal retrieval, and generation tasks, while showing robustness to client heterogeneity, modality corruption, and missing modalities.

Critical weaknesses
  1. The synergy term does not introduce a new gradient direction. When the hinge is active, excluding core regularization, $ \nabla_C\bigl(\ell_{\mathrm{joint}}+\lambda_c\mathcal L_{\mathrm{syn}}\bigr) =(1+\lambda_c)\nabla_C\ell_{\mathrm{joint}} $. Because the reference is detached, the term rescales the joint-task gradient without changing its direction.

  2. The isolated-branch references may become less informative during training. Disabling one update in a jointly adapted model may degrade the resulting branch’s performance. If both reference losses increase, the joint model can satisfy the margin against an increasingly weak comparator. Stop-gradient does not prevent this indirect deterioration, and branch-loss trajectories are not provided to assess reference quality.

  3. The coordination score measures agreement within each client and can assign identical scores to clients with opposite update directions, limiting its interpretation as global update quality.

Questions for authors
  1. Could the authors compare against task-loss-only training with an appropriately tuned learning rate? Since the detached-reference synergy term preserves the raw joint-task gradient direction, this would help establish its benefit beyond conditional gradient rescaling.

  2. Do the isolated-branch references deteriorate as the modules co-adapt, potentially making the synergy margin easier to satisfy?

  3. How does the same shared-core model perform with ordinary sample-weighted averaging and with coordination and drift weighting removed separately? These comparisons would help establish the incremental value of the proposed quality-aware aggregation.

  4. For the “w/o Shared Latent State” ablation, please clarify the exact independent-core configuration, including core dimensions, parameter-budget matching, and whether the SVD bases remain unchanged. How are the coordination score, normalized drift, and aggregation weights defined for the two independent states?

2 comments
AuthorsOpenReview ↗

Response to Reviewer xkiF (W1, W2, Q1, Q2)

W1 & Q1 — Gradient Interpretation and Practical Value of Synergy Loss

We thank the reviewer for the careful analysis of the gradient formulation. We confirm that, when the hinge term is active and the isolated-branch references are detached through stop-gradient, Synergy Loss conditionally rescales the joint task gradient without changing its direction.

We therefore clarify its intended role: Synergy Loss dynamically adjusts the weight of the joint task objective according to the performance of the perception-only and reasoning-only branches. Importantly, the central contribution of FedCORE lies in generating and aggregating heterogeneous module updates through a shared latent state, while Synergy Loss serves as a complementary optimization mechanism.

To evaluate its practical contribution, we compare full FedCORE against a learning-rate-tuned Task-only baseline and a variant without Synergy Loss.

MethodToys (Acc)DY (AUC)QB (R@5)Flickr30k (ROUGE-L)SemArt (CLIP-S)
Full FedCORE81.67 ± 0.0577.72 ± 0.3295.28 ± 0.6133.21 ± 0.7673.26 ± 0.21
Task-only (LR-tuned)81.29 ± 0.9876.54 ± 0.3894.78 ± 1.1732.32 ± 0.4772.97 ± 0.67
w/o Synergy Loss80.44 ± 0.2576.28 ± 0.1293.58 ± 0.8731.98 ± 0.4671.98 ± 0.57

Full FedCORE achieves the highest reported mean performance across all five tasks. These results demonstrate the empirical benefit of conditional task reweighting under the evaluated settings.

We have clarified the definition of Synergy Loss in Sec. 4.3 and added its gradient interpretation, the learning-rate-tuned Task-only comparison, and the ablation results in Appendix A.6.1 and Table 5.

W2 & Q2 — Reliability of Isolated-Branch References

We appreciate the reviewer's concern regarding the evolution of isolated-branch reference values during training.

To investigate this behavior, we track the task cross-entropy losses of the joint, perception-only, and reasoning-only branches throughout training, as illustrated in the supplementary figure (Fig. 10 in our revision version). All three losses decrease substantially during the early training stage. Although the two isolated-branch losses exhibit slight increases in later stages, neither shows a sustained upward trend.

These trajectories characterize the dynamic behavior of the references used by Synergy Loss. Rather than being fixed constants, they provide evolving performance references throughout model adaptation. Since the reference-branch losses are detached through stop-gradient, they contribute to evaluating and reweighting the joint objective without being directly optimized through this loss term.

Meanwhile, FedCORE's shared latent state remains responsible for coupling module updates, and its structural consistency under federated aggregation follows from the shared parameterization and the corresponding theoretical properties.

We have added the loss trajectories and their analysis in Appendix A.6.2 and Figure 10.

AuthorsOpenReview ↗

Response to Reviewer xkiF (W3, Q3, Q4)

Q3 — Effectiveness of Quality-Aware Aggregation

To isolate the contribution of quality-aware aggregation, we retain the same shared-core architecture and compare four aggregation strategies: standard sample-size weighting, coordination-only weighting, drift-only weighting, and the full quality-aware aggregation mechanism.

Aggregation StrategyToys (Acc)DY (AUC)QB (R@5)Flickr30k (ROUGE-L)SemArt (CLIP-S)
Sample-size weighting79.35 ± 0.3875.86 ± 0.2992.87 ± 0.2231.62 ± 0.4570.57 ± 0.15
Coordination-only80.55 ± 0.3676.32 ± 0.2594.35 ± 0.2132.07 ± 0.5371.91 ± 0.19
Drift-only80.69 ± 0.4176.34 ± 0.2894.02 ± 0.2232.05 ± 0.6571.45 ± 0.16
Full quality-aware aggregation81.67 ± 0.0577.72 ± 0.3295.28 ± 0.6133.21 ± 0.7673.26 ± 0.21

Across all five tasks, either coordination-based or drift-based weighting improves the reported mean performance over sample-size weighting alone. Combining both signals consistently yields the highest mean performance.

These findings support the complementary roles of coordination and drift as auxiliary signals for aggregating the shared latent core. We have included these comparisons in Table 3 of Sec. 5.3.

Q4 — Configuration of "w/o Shared Latent State"

We thank the reviewer for requesting clarification of this ablation setting.

The parameter-budget-matched Independent Core baseline employs two separate 11×1111\times11 cores, CkpC_k^p and CkrC_k^r, containing 242 trainable parameters in total. In contrast, FedCORE employs a single shared 16×1616\times16 core containing 256 trainable parameters. Both configurations retain the same fixed SVD bases.

Independent Core achieves an accuracy of 79.77±0.33%79.77\pm0.33\%, whereas FedCORE achieves 81.72±0.26%81.72\pm0.26\%. Thus, even under approximately matched adaptation parameter budgets, the shared-core design achieves higher accuracy.

We have clarified the core dimensions, trainable parameter counts, fixed SVD bases, and aggregation settings of Independent Core in Sec. 5.3 and Table 2. Its pairing evaluation protocol is further described in Appendix A.3.4.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary-Part1

We thank the Area Chair and the reviewers for their thoughtful assessment and constructive feedback. This summary highlights the paper’s contributions, the reviewers’ recognition of its strengths, and how the rebuttal addresses the main concerns.

Significance and strengths recognized by the reviewers

The reviews recognize the importance of coordinating multimodal perception and graph reasoning in federated learning. We appreciate all three reviewers' positive assessments of our work. xkiF recognizes the significance of coordinated perception-reasoning adaptation, the shared-core design, and the performance across tasks (Strengths S1–S3). saSH appreciates the clear motivation and strong experimental performance, highlighting the communication efficiency of transmitting only 512 scalars per round (S1–S2). wbnN recognizes cross-client module compatibility as a useful and reusable research perspective, explicitly identifying this framing as the main point in favor of acceptance, and also appreciates the exact aggregation property and communication scalability of the shared core (S1–S2).

Our central contribution is to formulate the issue of cross-module composability and address it through a single shared latent core that generates updates for heterogeneous perception and reasoning modules. With fixed linear decoders, core aggregation preserves a common coupled parameterization (Propositions A.2–A.3). The rebuttal provides complementary empirical evidence: under approximately matched parameter budgets, FedCORE improves both matched- and mismatched-pair performance over Independent Core, reducing the pairing gap from 8.1 to 6.4 percentage points. Across the five reported tasks, full FedCORE also achieves higher mean performance than both the learning-rate-tuned Task-only baseline and the variant without Synergy Loss. These results support the proposed coupling design and demonstrate the practical contributions of the accompanying optimization mechanisms.

AuthorsOpenReview ↗

Summary-Part2

Main concerns and our responses

1. Role of Synergy Loss and the source of its gains

Related concerns: xkiF W1–W2, Q1–Q2; saSH W1, Q1.

We accept both reviewers' gradient analysis and clarify that Synergy Loss performs conditional gradient reweighting, serving as an auxiliary optimization component. We added a learning-rate-tuned Task-only baseline, a comparison without Synergy Loss, and branch-loss trajectories. Full FedCORE achieves higher reported means across all five tasks (Appendices A.6.1–A.6.2, Table 5, Figure 10). Our gradient interpretation agrees with both reviewers' analysis, while the additional experiments address the questions about practical benefits and reference-branch reliability.

2. Contribution of the shared core beyond parameter count, warm-up, and task heads

Related concerns: xkiF Q4; saSH W2, Q2; wbnN W2, Q2.

We clarified the scope of parameter accounting and provided an approximately parameter-matched Independent Core comparison: 242 parameters versus FedCORE's 256 parameters, with FedCORE achieving higher means across all five tasks. The Warm-up + Zero-Core control further shows that trainable core adaptation improves performance by 2.53–10.67 percentage points across four datasets (Tables 2 and 6). These experiments directly address all three reviewers' questions about the sources of the performance gains.

3. Pairing-gap evidence and interpretation of coordination metrics

Related concerns: saSH W2, Q3; wbnN W1, Q1; xkiF W3, Q3.

We added pairing statistics over seven random seeds and clarified the evaluation protocol. The pairing gaps of Independent Joint LoRA, Module-wise FedAvg, and FedCORE are 25.5, 11.4, and 6.4 percentage points, respectively. Compared with the parameter-matched Independent Core baseline, FedCORE improves mismatched-pair accuracy by 5.2 percentage points and reduces the pairing gap by 1.7 percentage points (Figures 3 and 9). We also accept xkiF's distinction regarding the scope of the coordination metric: it measures within-client module agreement rather than cross-client gradient consensus. Additional aggregation-weight ablations support the complementary roles of coordination and drift signals (Table 3).

4. Experimental consistency, adaptation scope, and relationship to prior work

Related concerns: wbnN Q3–Q5.

We reran the robustness experiments and updated Figure 7 to correct its inconsistency with Table 1. We clarified that the CLIP backbone is frozen and that adaptation applies to the projection and fusion layers and the GNN. We also added a discussion of LoRA-XS, explaining the relationship between the single-module update subspaces and identifying cross-module core sharing as the key distinction. These revisions address wbnN's questions about experimental comparability, implementation scope, and the positioning of our contribution.

Reviewer feedback following the rebuttal

Following the rebuttal, wbnN increased their rating to 3. saSH thanked us for the detailed responses and additional experiments, appreciated our efforts to address their concerns, and maintained a rating of 3. All three reviewers currently rate the paper as Weak acceptance.

We believe that these revisions address the reviewers' substantive concerns through clearer definitions, controlled comparisons, multi-seed analyses, and corrected experimental results. Taken together, the evidence supports our central claim: a shared latent state helps improve the compatibility of multimodal perception and graph-reasoning updates under federated aggregation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.