- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FedMAGE for multimodal federated learning, where clients may differ in their modalities, tasks, and model architectures. Rather than aggregating incompatible model parameters, FedMAGE represents each client model as a typed computational graph and uses a graph hypernetwork to generate its parameters. The authors also introduce Type-aware Decoding (TAD) to account for differences in operator types and parameter structures, along with a learnable client embedding to capture client-specific information.
The method is evaluated on three benchmarks for image-text retrieval, multimodal sentiment recognition, and RGB-D scene recognition. The experiments compare FedMAGE with naive, personalized, and multimodal federated learning baselines, and examine the effect of TAD, varying levels of heterogeneity, communication time, and generalization to deeper variants within the same architecture families, specifically CNN-3 to CNN-5 and ViT-3 to ViT-5.
Critical strengths
-
Practical relevance. The paper considers architecture heterogeneity in multimodal federated learning, where clients may use different modalities, tasks, and model architectures. FedMAGE represents client models as typed computational graphs and generates client-specific parameters without requiring a shared parameter layout. This avoids the need to directly aggregate parameters from structurally different models. The method is evaluated on three benchmarks covering image-text retrieval, sentiment recognition, and RGB-D scene recognition.
-
TAD ablation results. Full TAD outperforms the shared MLP by 4.0, 5.8, and 4.6 points on the three tasks, respectively, and performs better than the other decoder variants. The comparisons also suggest that operator-aware decoding offers benefits beyond using separate decoders or type-specific heads, and that normalization further improves performance.
Critical weaknesses
-
The incremental contribution over related hypernetwork-based methods is unclear. FedMAGE uses a graph hypernetwork to generate parameters for heterogeneous client models. Related work has already explored architecture heterogeneity in federated learning (HAFL-GHN) and hypernetwork-based approaches to multimodal federated learning (FedMBridge). TAD appears to be the main methodological addition. While the ablations show that it performs better than the tested alternatives, they do not rule out the possibility that some of the gains come from the decoder's increased capacity. A comparison with a parameter-matched, operator-agnostic decoder would help isolate the effect of type-aware decoding. The lack of direct comparisons with HAFL-GHN and FedMBridge also makes it difficult to assess the advance over prior work.
-
Some of the paper's main claims are not directly tested. The paper aims to handle task, modality, and architecture heterogeneity jointly, but the three benchmarks appear to be evaluated separately. I could not find an experiment where clients with different task objectives participate in the same federation. The stress test also changes several aspects of heterogeneity at once, making it difficult to separate the effects of modality and architecture differences. For architecture generalization, the experiments move from CNN-3 to CNN-5 and from ViT-3 to ViT-5. This primarily tests depth extrapolation within known architecture families, providing limited evidence of generalization across architecture families or to substantially different operator configurations. Xavier initialization provides only a limited reference point for this claim; comparison with a trained GHN or another parameter-generation method would be more informative.
Questions for authors
-
Difference from existing hypernetwork-based methods. I would like to better understand what distinguishes FedMAGE from HAFL-GHN and FedMBridge beyond the use of TAD. For example, does FedMAGE introduce a different way of representing client architectures, conditioning the hypernetwork on client-specific information, or training the model across clients? Or is the main contribution the decoder design itself? A more direct comparison with these methods would help answer this question. I am also unsure how much of the improvement can be attributed specifically to type-aware decoding, since the current ablation does not appear to control for decoder capacity. Comparing TAD with a standard decoder of similar parameter count would help determine whether the gains come from using operator-type information or from the additional capacity.
-
How client embeddings are used in Table 4. The architecture generalization experiment changes CNN-3 to CNN-5 and ViT-3 to ViT-5 while keeping the trained generator fixed, but I could not find a clear explanation of how the client embedding is handled in this process. Is the embedding learned for the original client retained when its architecture is changed? If so, does the experiment test whether the generator can produce useful parameters for a new architecture given an already learned client representation? Alternatively, is a new embedding initialized for the changed architecture, and if so, is it updated in any way? Since the generated parameters depend on both the architecture graph and the client embedding, this detail seems important for interpreting what Table 4 actually demonstrates.
-
Whether task heterogeneity is evaluated jointly. I could not tell from the experiments whether clients performing different tasks participate in the same federation. Are the image-text retrieval, sentiment recognition, and RGB-D tasks trained separately, with a different federation for each benchmark, or does a single hypernetwork support clients with different task objectives at the same time? The distinction matters because evaluating several tasks separately does not, by itself, show that the method can accommodate task heterogeneity within one federation. If the paper does include such an experiment, could the authors point to the relevant results? If not, what evidence supports the claim that FedMAGE handles task heterogeneity jointly with modality and architecture heterogeneity?
8 comments
Response to Reviewer PpCc - Part 2
FedMAGE outperforms both FedMBridge and HAFL-GHN on all six evaluation metrics**.** Compared with FedMBridge:
- Image–text retrieval: Flickr30K mR increases from 78.5% to 81.4%, an improvement of 2.9%.
- Sentiment recognition: CMU-MOSEI Acc-7, Acc-2, and F1 improve by 3.8%, 4.1%, and 3.1%, respectively, covering both fine-grained and binary sentiment classification.
- Scene recognition: NYU-Depth-v2 Top-1 and F1 improve by 3.6% and 1.8%, respectively.
These above comparisons establish the predictive advantage of the complete FedMAGE method across the three benchmarks under the same client architectures and training schedules.
1.3 Empirical Evidence and Theoretical Rationale for Type-aware Decoding
The existing ablations from Table 2 examine complementary aspects of the decoding design. Full TAD outperforms Shared MLP by 4.0%, 5.8%, and 4.6%, and Separate Decoders by 2.3%, 3.8%, and 2.9%, on Flickr30K mR, CMU-MOSEI Acc-7, and NYU-Depth-v2 Top-1, respectively. Furthermore, adding normalization to “Query + Heads, w/o Norm” improves these metrics by 1.8%, 2.6%, and 2.4%. These results consistently support the complete design, which combines shared intermediate generation features, operator-specific output mappings, and normalization.
Structural distinction at a matched parameter count. Matching parameter counts controls the number of trainable coefficients; operator-specific routing determines how those coefficients are shared and updated. To analyze this distinction, consider a shared decoder with the same total parameter count, the same query inputs (including operator-type information) and the same normalization. TAD explicitly assigns each operator type to its corresponding output head, while retaining a shared trunk . This assignment imposes a specific parameter-sharing structure that does not follow from parameter count alone.
This property can be expressed through the server objective in Eq. (12). Let denote the component associated with generated parameters of operator type , with the local-training targets held fixed by the stop-gradient operator. The objective decomposes as:
.
Writing for the trainable parameters of output head , the decoding structure in Eq. (9) gives , and , . Thus, the reconstruction loss associated with one operator type directly updates its corresponding output head without directly changing the heads assigned to other types. For a shared output head , the gradient instead combines contributions from all operator types:
.
When different operator types favor opposing changes to shared output parameters, these contributions can interfere. TAD places their direct output-head updates in separate parameter blocks. Increasing the width of a shared decoder changes its parameter budget, but does not itself impose this operator-specific separation.
Complementary roles of specialization, sharing, and normalization. The graph encoder and continue to receive training signals from different operator types, supporting shared intermediate representations. The specialized heads provide dedicated output adaptation for operators with different parameter structures. Applying the same operator-specific normalization to both decoder designs controls generated parameter statistics while preserving this difference in output-head dependencies.
A shared MLP can learn type-dependent mappings from its inputs; TAD builds the corresponding specialization directly into the decoder architecture. This provides an explicit inductive bias for heterogeneous parameter generation. Table 2 supports the effectiveness of the complete design, while the analysis identifies its structural contribution under matched parameter count and normalization: shared intermediate generation combined with operator-specific output adaptation.
Table 2 in main-text provides the following evidence for this design within FedMAGE:
| Decoder variant | Flickr30K (mR↑) | MOSEI (Acc-7↑) | NYU-Depth-v2 (Top-1↑) |
|---|---|---|---|
| Shared MLP | 77.4 | 51.0 | 77.8 |
| Query + Decoder | 79.5 | 53.4 | 79.8 |
| Separate Decoders | 79.1 | 53.0 | 79.5 |
| Type-specific Heads | 78.0 | 52.4 | 79.1 |
| Query + Heads, without normalization | 79.6 | 54.2 | 80.0 |
| Full TAD | 81.4 | 56.8 | 82.4 |
Response to Reviewer PpCc - Part 3
Question 2. Client Embeddings in the Architecture Generalization Experiment
We thank you for highlighting this important detail. The embedding learned for the original client is retained and frozen when its architecture changes. The trained generator is also frozen, and the generated model is evaluated directly without additional local training or fine-tuning. Table 4 (main-text) therefore evaluates parameter generation for an unseen, deeper architecture of an existing client, conditioned on that client’s already learned representation.
2.1 How the Client Embedding is Handled
Let and denote the generator parameters and client embedding obtained after federated training. When changing CNN-3 to CNN-5 or ViT-3 to ViT-5, we replace the original architecture graph with the corresponding deeper graph . The parameters of the new model are generated as . The evaluation protocol is as follows:
| Component | Treatment in Table 4 |
|---|---|
| Architecture graph | Replace with the deeper architecture graph . |
| Generator parameters | Keep the trained parameters fixed. |
| Client embedding | Retain the original client’s learned embedding and keep it fixed. |
| New model parameters | Generate from and evaluate directly on the test set. |
No new client embedding is initialized, and the embedding-update rule in Eq. (14) is not applied during this evaluation. Thus, “zero-shot” refers specifically to generating and evaluating parameters for the changed architecture without adaptation to that architecture.
2.2 Why a Fixed Client Embedding Supports a Changed Architecture
The client embedding and architecture graph provide complementary conditioning inputs. The embedding supplies the client-specific context learned during federated training, while the graph explicitly specifies the target model’s operators, computational dependencies, and parameter shapes. The dimensionality of is independent of the number of layers in the client model.
Keeping fixed therefore preserves the learned client conditioning while allowing parameter generation to respond to the new architecture. In particular, node initialization remains , but the node attributes and message-passing structure are now determined by . The resulting graph representations enter the parameter query in Eq. (8):
.
Although the client embedding is unchanged, the graph-dependent terms can change with the added layers and modified topology. The shared trunk and type-specific heads then generate parameters for the parametric nodes of including the newly introduced layers. This is possible because the graph encoder and decoding functions are shared across nodes and operate within the predefined operator vocabulary. Increasing the depth introduces additional instances and connections of supported operators; it does not require a new client embedding or a separate decoder for the deeper architecture.
2.3 What Table 4 Demonstrates
Table 4 (main-text) evaluates within-family depth generalization under a fixed, learned client representation. It tests whether the trained client-conditioned generator can produce useful parameters when an existing client changes its architecture from three to five layers.
The existing results support this interpretation:
- CNN-3 → CNN-5: FedMAGE achieves 59.5%, 35.6%, and 60.2% on Flickr30K mR, CMU-MOSEI Acc-7, and NYU-Depth-v2 Top-1, respectively. These exceed Xavier initialization by 32.7%, 17.2%, and 31.2%.
- ViT-3 → ViT-5: FedMAGE achieves 56.4%, 34.0%, and 64.3%, exceeding Xavier initialization by 31.1%, 14.5%, and 36.4%, respectively.
These scores are obtained with both and frozen and without fine-tuning the generated models. They show that the learned generator and client representation can be reused together to produce useful parameters for a previously unseen deeper architecture. The architectural generalization demonstrated in Table 4 is conditional on an existing client’s learned representation, which remains fixed throughout the architecture change.
Table 4 (main-text). Zero-shot generalization to unseen deeper client architectures.
| Scenario | Initialization | Flickr30K mR ↑ | CMU-MOSEI Acc-7 ↑ | NYU-Depth-v2 Top-1 ↑ |
|---|---|---|---|---|
| No switch | FedMAGE | 81.4 | 56.8 | 82.4 |
| CNN-3 → CNN-5 | Xavier init | 26.8 | 18.4 | 29.0 |
| CNN-3 → CNN-5 | FedMAGE | 59.5 | 35.6 | 60.2 |
| ViT-3 → ViT-5 | Xavier init | 25.3 | 19.5 | 27.9 |
| ViT-3 → ViT-5 | FedMAGE | 56.4 | 34.0 | 64.3 |
Response to Reviewer PpCc - Part 1
We thank the reviewer PpCc for taking the time to carefully evaluate our paper and for the constructive comments. Your suggestions have helped us clarify the methodological contribution, strengthen the comparison with related methods, and improve the interpretation of our experimental evidence.
Your comments are highly professional. To address it, we elaborate on the advantages of our method below and supplement comparative experiments with HAFL-GHN [1] and FedMBridge [2]. Please find our response below, which we hope addresses your concerns regarding our manuscript.
1.1 Methodological Distinctions and Advantages
The main methodological contribution of FedMAGE is the decoding design in Section 3.2, together with client conditioning during both architecture encoding and parameter generation. Representing client architectures as computational graphs follows the graph-hypernetwork framework. The distinctions concern how client information enters this representation and how the resulting features are converted into model parameters:
Regarding federated training, FedMAGE and FedMBridge share the broad procedure of generating client parameters on the server, training client models locally, and using returned updates to train the generator. HAFL-GHN instead trains local copies of the graph hypernetwork and aggregates their parameters. Therefore, FedMAGE’s contribution is concentrated in its client-conditioned encoding and structured decoding within the server-side generation framework.
1.2 Direct Comparison under the Same Client Architectures and Training Budgets
We have completed direct comparisons with HAFL-GHN and FedMBridge using our reimplementations of their published methods, as their code is not publicly available. All methods use the same per-client architectures, data partitions, modality assignments, participation schedules, communication rounds, and local minibatch-update budgets. Each baseline retains its original federated optimization procedure.
Table R1. Comparison with graph-hypernetwork-based federated learning methods. All scores are reported as percentages.