- Rating
- 2
- Confidence
- 4
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies the modality collaspe in multimodal federated learning, where a dominant modality takes over and the weaker modality's encoder loses representational capacity. The authors point out that traditional Fedavg makes this worse because weak modality update disagree across clients. They propose FedSBR that split encoder feature into shared and private coordinates, then combines three components: capacity-preserving regularization, gated dominant-to-weak distillation and federated class-prototype anchoring. Experiments on four multimodal datasets illustrate the effectiveness compared to other baselines.
Critical strengths
-
The paper tackles a timely and practical problem which is modality collaspe in multimodal federated learning.
-
The proposed method FedSBR improves consistently over FedAvg and other baselines on four datasets under both IID and non-IID settings. Further diagnostics and component ablations prove the effectiveness of the method.
Critical weaknesses
-
The motivating diagnostics in Figure 1 are not explained clearly. The figure presents semantic decodability and representation rank without defining them or describing how they are measured. Representation rank appears later as the participation rank in Eq . (6) and how semantic decodability is measured is never stated.
-
The "two coupled" levels in L80-83 are unclear. The text says FedSBR addresses collaspe at "two coupled levels" but then lists three components (CPR, DGT, FSA) without naming two levels or mapping components to them. Please state the two levels explicitly and explain how they are coupled.
-
The methodology is quite hard to follow. Section 3 introduces many interacting terms, gates and thresholds and Figure 2 is not tied closely to the text.
-
The method has many hyperparameters and its sensitvitiy of those hyperparameters is not analyzed yet. This raises concerns how to tune the FedSBR method on each dataset.
Questions for authors
-
How does FedSBR's communication cost compare with other baselines? Could the authors give the comparison against FedAvg, FedProto ?
-
How is the scalability of FedSBR? The main experiments use a small number of clients with 10 clients. How does it perform with such more clients ?
6 comments
Questions
Q1. Communication relative to FedAvg and FedProto
Compared with FedAvg, our exchange includes additional model parameters and class summaries. Appendix C.5 gives additional parameters and at most uploaded summary scalars per client. For two modalities, width 512 and shared width 256, the FP32 payloads are:
| Dataset | Extra model Up MiB | Extra model Down MiB | Summary Up KiB | Prototype vectors Down KiB |
|---|---|---|---|---|
| CREMA-D | 4.01 | 4.01 | 6.05 | 6.00 |
| ModelNet40 | 4.04 | 4.04 | 40.31 | 40.00 |
The maximum upload assumes every class is represented on the client. The last column counts prototype vectors, with supports and masks accounted for separately. Relative to FedAvg's model payload P in each direction, our exchange adds the model increment above and the summaries.
The original FedProto (Tan et al., 2022) exchanges class prototypes while keeping model parameters local. For prototype width , a full class bank contains values, or bytes in FP32 per direction, excluding metadata. Our exchange also includes the model parameters needed to train a shared global predictor. For these comparisons, we count tensor payloads, exclude transport headers and sum per-client unicast downloads.
Q2. Scaling beyond ten clients
In fact, our existing client-count sweep covers 5 to 100 clients (Table 12), with a single seed and a short training budget. With 100 clients, five participants per round and five rounds, only 22.6% of clients are expected to be selected at least once (C.11). The comparison below holds expected participation per client fixed to separate scale from training exposure.
With 50% participation and 100 rounds, each client has 50 expected selections. Keeping total data fixed, using two local epochs and retaining incomplete batches also matches the expected number of example presentations per round.
| Clients | Mean samples | Method | Fusion ↑ | Visual ↑ | Rank ↑ | GPU h |
|---|---|---|---|---|---|---|
| 10 | 580.3 | FedAvg | 45.39 | 22.09 | 2.59 | 0.46 |
| FedAdam | 49.72 | 27.12 | 2.20 | 0.68 | ||
| FedSBR | 50.39 | 28.30 | 4.08 | 0.60 | ||
| 50 | 116.1 | FedAvg | 43.12 | 21.88 | 2.46 | 0.52 |
| FedAdam | 47.84 | 25.74 | 2.16 | 0.74 | ||
| FedSBR | 48.06 | 27.76 | 3.91 | 0.75 | ||
| 100 | 58.0 | FedAvg | 41.96 | 21.45 | 2.32 | 0.64 |
| FedAdam | 45.35 | 24.86 | 2.08 | 0.87 | ||
| FedSBR | 45.10 | 26.71 | 3.74 | 0.96 |
Across client counts, we keep CREMA-D's global train/validation/test split fixed. Within each client count, all methods share the same client partition and sampling schedule. Our settings are alpha = 0.5 and three paired seeds. Mean samples is 5,803 divided by client count.
To Dear Reviewer Xaj3
Thank you for the time and care you have devoted to reviewing our work. We have updated the rebuttal version of our manuscript to clarify the diagnostic definitions and method presentation. The revision also includes loss-weight sensitivity, communication-payload accounting, and client-scaling analyses in Appendix Sections C.16–C.18.
For ease of reference:
- Light yellow: revisions addressing Reviewer vtxE.
- Light blue: revisions addressing Reviewer 54zV.
- Light purple: revisions addressing Reviewer Xaj3.
Changes to figures and tables are marked in their captions. For points raised by more than one reviewer, the color indicates the primary comment addressed.
Thank you for the detailed comments on clarity and tuning.
Weaknesses
By rank, we mean the participation rank in Eq. (6). For a centered feature matrix H with b samples,
We assign rank zero when C=0. Uniform rescaling leaves the rank unchanged.
Semantic decodability in Fig. 1 is pathway classification accuracy on the validation set. We retain the target modality's input, set the other modality's input to zero (B.2), and run the complete encoder, fusion and prediction pipeline g(φ(⋅)). Predictions come from the fusion classifier. Our unimodal heads provide training supervision and gate statistics. Pathway accuracy measures how well our fused predictor uses the retained modality, while participation rank measures feature geometry. We will add this definition and evaluation protocol to the caption of Fig. 1.
Our design acts at two levels: local capacity and predictive learning through CPR and DGT, and cross-client semantic coordination through FSA.
Both levels share the gate in Eq. (7), which weights local capacity, transfer and FSA's weak-pathway semantic term. For FSA, we compute the prototypes from the shared features learned locally. In our representation, "private" denotes modality-specific coordinates, and both coordinate blocks are aggregated globally (Sec. 3.2).
The local path in our Fig. 2(b) follows Eq. (5): encoding, shared/private projection, reconstruction and prediction. The side paths apply CPR (Eq. (8)), DGT (Eq. (9)) and FSA (Eqs. (10)-(11)). Capacity, distillation and weak-pathway semantic terms are gated, while reconstruction and direct unimodal/shared supervision remain active.
For the outer loop in Fig. 2(a), we broadcast the model and prototypes, train locally, and upload parameters and class summaries (Algorithm 1, B.3). We will link these steps to their equations in Section 3 of our manuscript.
For CREMA-D, ModelNet40 and CrisisMMD, our implementation provides common defaults (Table 5). The CPR coefficients are λv=0.05, λo=0.01 and λr=0.05. Across all four datasets, we use the same gate thresholds, qmin=0.50 and δmin=0.10. We will include these omitted thresholds and make the dataset-specific settings explicit alongside the shared defaults. Checkpoints are selected by validation fusion accuracy, or mean validation AUROC for MIMIC-IV (B.1).
Regarding loss-weight sensitivity, the comparison below scales the coefficients of CPR, DGT or FSA by 0.5 or 2, one module at a time. Internal ratios and other settings remain fixed. Default reference results are from Table 6.