acceptodds
Desk rejected at ICLR 2027

Federated Shared–Private Basis Reallocation for Multimodal Representation Collapse

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal federated learning can overuse a dominant modality, while heterogeneous client updates suppress effective directions in weaker pathways. Existing approaches regulate modality selection, optimization, or distillation, but leave weak encoder capacity unprotected during aggregation. We introduce Federated Shared and Private Basis Reallocation (FedSBR), an encoder-level framework that separates shared semantic coordinates from private residual coordinates. FedSBR preserves weak-pathway capacity through collapse-aware regularization, transfers supervision only when loss, rank, and teacher reliability indicate dominance, and coordinates class semantics through reliability-weighted federated prototypes. Identity-preserving initialization reconstructs features without changing the initial predictor. Experiments on CREMA-D, ModelNet40, CrisisMMD, and MIMIC-IV under IID and Dirichlet non-IID partitions show improvements over FedAvg and competitive fusion performance. Pathway accuracy, participation rank, modality contribution, and class-conditional analyses collectively support the proposed mechanism under federated heterogeneity.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal Federated Learning, Modality Collapse, Basis Reallocation, Prototype Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Xaj3OpenReview ↗
Rating
2
Confidence
4
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper studies the modality collaspe in multimodal federated learning, where a dominant modality takes over and the weaker modality's encoder loses representational capacity. The authors point out that traditional Fedavg makes this worse because weak modality update disagree across clients. They propose FedSBR that split encoder feature into shared and private coordinates, then combines three components: capacity-preserving regularization, gated dominant-to-weak distillation and federated class-prototype anchoring. Experiments on four multimodal datasets illustrate the effectiveness compared to other baselines.

Critical strengths
  • The paper tackles a timely and practical problem which is modality collaspe in multimodal federated learning.

  • The proposed method FedSBR improves consistently over FedAvg and other baselines on four datasets under both IID and non-IID settings. Further diagnostics and component ablations prove the effectiveness of the method.

Critical weaknesses
  • The motivating diagnostics in Figure 1 are not explained clearly. The figure presents semantic decodability and representation rank without defining them or describing how they are measured. Representation rank appears later as the participation rank in Eq . (6) and how semantic decodability is measured is never stated.

  • The "two coupled" levels in L80-83 are unclear. The text says FedSBR addresses collaspe at "two coupled levels" but then lists three components (CPR, DGT, FSA) without naming two levels or mapping components to them. Please state the two levels explicitly and explain how they are coupled.

  • The methodology is quite hard to follow. Section 3 introduces many interacting terms, gates and thresholds and Figure 2 is not tied closely to the text.

  • The method has many hyperparameters and its sensitvitiy of those hyperparameters is not analyzed yet. This raises concerns how to tune the FedSBR method on each dataset.

Questions for authors
  • How does FedSBR's communication cost compare with other baselines? Could the authors give the comparison against FedAvg, FedProto ?

  • How is the scalability of FedSBR? The main experiments use a small number of clients with 10 clients. How does it perform with such more clients ?

6 comments

Thank you for the detailed comments on clarity and tuning.

Weaknesses

W1. Definitions of the motivating diagnostics

By rank, we mean the participation rank in Eq. (6). For a centered feature matrix HH with bb samples,

r(C)=tr⁡(C)2tr⁡(C2),C=H⊤Hb.r(C)=\frac{\operatorname{tr}(C)^2}{\operatorname{tr}(C^2)},\qquad C=\frac{H^\top H}{b}.

We assign rank zero when C=0C=0. Uniform rescaling leaves the rank unchanged.

Semantic decodability in Fig. 1 is pathway classification accuracy on the validation set. We retain the target modality's input, set the other modality's input to zero (B.2), and run the complete encoder, fusion and prediction pipeline g(φ(⋅))g(\varphi(\cdot)). Predictions come from the fusion classifier. Our unimodal heads provide training supervision and gate statistics. Pathway accuracy measures how well our fused predictor uses the retained modality, while participation rank measures feature geometry. We will add this definition and evaluation protocol to the caption of Fig. 1.

W2. Two levels and three components

Our design acts at two levels: local capacity and predictive learning through CPR and DGT, and cross-client semantic coordination through FSA.

Both levels share the gate in Eq. (7), which weights local capacity, transfer and FSA's weak-pathway semantic term. For FSA, we compute the prototypes from the shared features learned locally. In our representation, "private" denotes modality-specific coordinates, and both coordinate blocks are aggregated globally (Sec. 3.2).

W3. Linking Fig. 2 to the method

The local path in our Fig. 2(b) follows Eq. (5): encoding, shared/private projection, reconstruction and prediction. The side paths apply CPR (Eq. (8)), DGT (Eq. (9)) and FSA (Eqs. (10)-(11)). Capacity, distillation and weak-pathway semantic terms are gated, while reconstruction and direct unimodal/shared supervision remain active.

For the outer loop in Fig. 2(a), we broadcast the model and prototypes, train locally, and upload parameters and class summaries (Algorithm 1, B.3). We will link these steps to their equations in Section 3 of our manuscript.

W4. Hyperparameter sensitivity and tuning

For CREMA-D, ModelNet40 and CrisisMMD, our implementation provides common defaults (Table 5). The CPR coefficients are λv=0.05\lambda_v=0.05, λo=0.01\lambda_o=0.01 and λr=0.05\lambda_r=0.05. Across all four datasets, we use the same gate thresholds, qmin⁡=0.50q_{\min}=0.50 and δmin⁡=0.10\delta_{\min}=0.10. We will include these omitted thresholds and make the dataset-specific settings explicit alongside the shared defaults. Checkpoints are selected by validation fusion accuracy, or mean validation AUROC for MIMIC-IV (B.1).

Regarding loss-weight sensitivity, the comparison below scales the coefficients of CPR, DGT or FSA by 0.5 or 2, one module at a time. Internal ratios and other settings remain fixed. Default reference results are from Table 6.

DatasetScaled lossFactorFusion ↑Weak ↑Rank ↑
CREMA-DDefault150.3928.304.08
CPR0.549.8227.173.81
CPR249.9628.724.24
DGT0.549.9027.754.02
DGT250.1228.594.13
FSA0.549.7127.934.01
FSA250.0828.424.11
ModelNet40Default184.2065.957.58
CPR0.583.4265.277.20
CPR283.9866.127.92
DGT0.582.3161.807.34
DGT284.0866.427.61
FSA0.583.2465.557.47
FSA284.3765.777.64

Questions

Q1. Communication relative to FedAvg and FedProto

Compared with FedAvg, our exchange includes additional model parameters and class summaries. Appendix C.5 gives 2Md2+Cds+C2Md^2+Cd_s+C additional parameters and at most C(ds+2)C(d_s+2) uploaded summary scalars per client. For two modalities, width 512 and shared width 256, the FP32 payloads are:

DatasetExtra model Up MiBExtra model Down MiBSummary Up KiBPrototype vectors Down KiB
CREMA-D4.014.016.056.00
ModelNet404.044.0440.3140.00

The maximum upload assumes every class is represented on the client. The last column counts prototype vectors, with supports and masks accounted for separately. Relative to FedAvg's model payload P in each direction, our exchange adds the model increment above and the summaries.

The original FedProto (Tan et al., 2022) exchanges class prototypes while keeping model parameters local. For prototype width dprotod_{\mathrm{proto}}, a full class bank contains CdprotoCd_{\mathrm{proto}} values, or 4Cdproto4Cd_{\mathrm{proto}} bytes in FP32 per direction, excluding metadata. Our exchange also includes the model parameters needed to train a shared global predictor. For these comparisons, we count tensor payloads, exclude transport headers and sum per-client unicast downloads.

Q2. Scaling beyond ten clients

In fact, our existing client-count sweep covers 5 to 100 clients (Table 12), with a single seed and a short training budget. With 100 clients, five participants per round and five rounds, only 22.6% of clients are expected to be selected at least once (C.11). The comparison below holds expected participation per client fixed to separate scale from training exposure.

With 50% participation and 100 rounds, each client has 50 expected selections. Keeping total data fixed, using two local epochs and retaining incomplete batches also matches the expected number of example presentations per round.

ClientsMean samplesMethodFusion ↑Visual ↑Rank ↑GPU h
10580.3FedAvg45.3922.092.590.46
FedAdam49.7227.122.200.68
FedSBR50.3928.304.080.60
50116.1FedAvg43.1221.882.460.52
FedAdam47.8425.742.160.74
FedSBR48.0627.763.910.75
10058.0FedAvg41.9621.452.320.64
FedAdam45.3524.862.080.87
FedSBR45.1026.713.740.96

Across client counts, we keep CREMA-D's global train/validation/test split fixed. Within each client count, all methods share the same client partition and sampling schedule. Our settings are alpha = 0.5 and three paired seeds. Mean samples is 5,803 divided by client count.

AuthorsOpenReview ↗

To Dear Reviewer Xaj3

Thank you for the time and care you have devoted to reviewing our work. We have updated the rebuttal version of our manuscript to clarify the diagnostic definitions and method presentation. The revision also includes loss-weight sensitivity, communication-payload accounting, and client-scaling analyses in Appendix Sections C.16–C.18.

For ease of reference:

  • Light yellow: revisions addressing Reviewer vtxE.
  • Light blue: revisions addressing Reviewer 54zV.
  • Light purple: revisions addressing Reviewer Xaj3.

Changes to figures and tables are marked in their captions. For points raised by more than one reviewer, the color indicates the primary comment addressed.

Reviewer_54zVOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates modality collapse in multimodal federated learning and identifies two key challenges: (i) dominant modalities over-occupy fusion directions during local optimization, and (ii) heterogeneous client updates suppress weak-pathway directions during server aggregation. The authors propose FedSBR, an encoder-level framework comprising shared–private basis reallocation, Capacity-Preserving Reallocation (CPR), Dominance-Gated Transfer (DGT), and Federated Semantic Anchoring (FSA). Experiments on four datasets under IID and non-IID partitions show that FedSBR improves over FedAvg and remains competitive with strong fusion baselines, with pathway- and class-level diagnostics supporting the proposed mechanism.

Critical strengths
  1. The proposed modules have clearly delineated roles.CPR protects the private coordinates of weak modalities locally through variance–covariance regularization and reconstruction loss, preventing the representation space from being compressed; DGT transfers supervision from the dominant modality to weak pathways only when the loss gap, rank deficit, and teacher confidence all satisfy the gating conditions, while always retaining direct unimodal supervision; FSA coordinates shared semantics and class coverage through quality-weighted federated class prototypes, without requiring any public proxy data.
  2. The experiments are relatively comprehensive.The paper conducts a broad comparison across multiple multimodal datasets and both IID and non-IID partitions, against a range of federated baselines and targeted controls, and further supplements this with ablations, convergence, and computational cost analyses. The experimental results show that FedSBR outperforms FedAvg in fusion performance and remains competitive with strong baselines, which demonstrates the effectiveness of the proposed method.
Critical weaknesses

1.The gate thresholds qmin and δmin are never assigned values; Table 5 lists ρ, τg, and τd, but omits these two. 2. The paper only provides a whole-module ablation; it does not empirically isolate which of the four CPR terms is responsible for the weak-pathway capacity gain.

Questions for authors

1.The gate thresholds qminand δminare not assigned values in the main text or in Table 5, and no sensitivity or ablation over them is reported. Could the authors state the values used in the main experiments, and report how sensitive DGT is to these two thresholds (e.g., a small sweep or a gate-on/off analysis)? 2.Table 3 ablates CPR only as a whole. Could the authors separately remove the variance floor, the covariance penalty, the shared–private orthogonality term, and the reconstruction term, and report the resulting fusion accuracy, weak-pathway accuracy, and participation rank?

2 comments

Thank you for the careful review and the specific ablations you suggest.

Weaknesses

W1. Missing threshold values

We omitted both thresholds from Table 5 in our manuscript. Their values and sensitivity are addressed in Q1.

W2. CPR is only ablated as a whole

Our existing ablation evaluates CPR as a whole. Removing it lowers CREMA-D weak accuracy from 28.30 to 22.75 and rank from 4.08 to 3.24 (Table 16). Q2 separates the four terms in a controlled ablation.

Questions

Q1. Threshold values and sensitivity

Our main experiments use qmin⁡=0.50q_{\min}=0.50 and δmin⁡=0.10\delta_{\min}=0.10 across all four datasets. We will add both values to Table 5.

For loss gap dd, relative rank deficit δr\delta^r and teacher confidence qq, our gate in Eq. (7) is

α=1[d>0, δr≥δmin⁡, q≥qmin⁡] gdgrgq.\alpha=\mathbb{1}\left[d>0,\ \delta^r\geq\delta_{\min},\ q\geq q_{\min}\right]\,g^d g^r g^q.

Teacher confidence qq is normalized relative to chance level 1/C1/C. Thus qmin⁡=0.50q_{\min}=0.50 corresponds to a mean maximum class probability of (1+1/C)/2(1+1/C)/2. The rank threshold requires a relative deficit of at least 10%, using Eq. (7)'s stabilized denominator. Rank determines eligibility, while loss and confidence modulate the weight. When the gate closes, we retain direct weak-pathway supervision, reconstruction and shared supervision.

We have added a threshold-sensitivity analysis table with one-at-a-time variations around the main setting and a loss-gap gate control. The settings and results are shown below. Pairs denote (qmin⁡,δmin⁡)(q_{\min},\delta_{\min}). Active is the percentage of available teacher-to-other-modality batch pairs passing the hard gate, including loss-gap failures in the denominator.

DatasetGate settingActive %Fusion ↑Weak ↑Rank ↑
CREMA-D(0.50, 0.10)32.450.3928.304.08
(0.30, 0.10)44.649.9828.354.10
(0.70, 0.10)18.249.5427.423.92
(0.50, 0.05)42.350.1228.514.17
(0.50, 0.20)20.749.8327.663.96
Loss-gap gate67.849.1227.624.18
ModelNet40(0.50, 0.10)38.684.2065.957.58
(0.30, 0.10)49.283.8265.617.55
(0.70, 0.10)25.483.3164.657.29
(0.50, 0.05)47.884.0966.087.66
(0.50, 0.20)27.983.7464.947.36
Loss-gap gate72.183.1564.397.61

For the loss-gap control, we set α=1(d>0)gd\alpha=\mathbf{1}(d>0)g^d and retain teacher selection, loss normalization and Eq. (9)'s separate confidence multiplier. Because the gate is shared, this control applies to CPR, distillation and the weak-pathway semantic penalties together.

Q2. Individual CPR terms

Following your suggestion, we have added a term-by-term ablation table in which one term from Eq. (8) is removed at a time, with DGT, FSA, the gate rule and the other coefficients fixed. Variance and covariance share λv\lambda_v, so each ablation removes only the corresponding summand. Cross-cov. denotes our shared-private orthogonality penalty.

DatasetVarianceCovarianceCross-cov.Recon.Fusion ↑Weak ↑Rank ↑
CREMA-D✓✓✓✓50.39 ± 0.9728.30 ± 1.414.08 ± 0.18
✗✓✓✓48.67 ± 1.2424.80 ± 1.623.35 ± 0.26
✓✗✓✓49.88 ± 1.0527.68 ± 1.493.77 ± 0.23
✓✓✗✓50.17 ± 1.0928.42 ± 1.384.13 ± 0.21
✓✓✓✗49.26 ± 1.1826.42 ± 1.573.83 ± 0.25
ModelNet40✓✓✓✓84.20 ± 0.6265.95 ± 1.047.58 ± 0.31
✗✓✓✓82.60 ± 0.8164.82 ± 1.226.65 ± 0.39
✓✗✓✓83.72 ± 0.7165.60 ± 1.136.96 ± 0.36
✓✓✗✓84.29 ± 0.6666.08 ± 1.097.44 ± 0.33
✓✓✓✗82.98 ± 0.7964.93 ± 1.197.11 ± 0.37
AuthorsOpenReview ↗

To Dear Reviewer 54zV

We appreciate your careful assessment and your specific suggestions on the gate thresholds and CPR components. We have revised the manuscript for the rebuttal phase to specify both thresholds and include the threshold-sensitivity study and individual CPR-term ablations in Appendix Sections C.14–C.15.

We mark the revisions as follows:

  • Light yellow: revisions addressing Reviewer vtxE.
  • Light blue: revisions addressing Reviewer 54zV.
  • Light purple: revisions addressing Reviewer Xaj3.

Figure and table updates are indicated by highlighted captions. For points raised by more than one reviewer, the color indicates the primary comment addressed.

Reviewer_vtxEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedSBR, which combines shared-private representation decomposition, weak-modality capacity preservation, and federated semantic alignment to mitigate modality representation collapse in multimodal federated learning.

Critical strengths
  1. The paper investigates modality collapse from the perspective of how federated aggregation affects weak-modality representations, with clear research motivation.

  2. The method combines capacity preservation, conditional knowledge distillation, and semantic prototype alignment to enable collaborative optimization of weak-modality representations.

  3. Experiments are conducted on four datasets, with additional analyses of modality accuracy, representation rank, and modality contributions to validate the effectiveness of the method.

Critical weaknesses
  1. The method combines several existing techniques, and the novelty of individual components compared with prior work needs further clarification.

  2. Under some heterogeneous data settings, the fusion performance does not outperform certain baselines, requiring further explanation of the relationship between weak-modality preservation and overall performance.

  3. The shared and private representation dimensions are allocated using a fixed ratio, lacking further analysis of different dimension allocation strategies.

Questions for authors

Please see above.

2 comments

Thank you for the insightful review!

Weaknesses

W1. Contribution beyond existing components

  • Our contribution couples local capacity preservation with cross-client semantic coordination. Clients can retain diverse features while encoding the same class differently.

  • CPR preserves private feature diversity, and DGT provides direct supervision and gated teacher guidance. FSA adds shared supervision and semantic constraints. The w/o-FSA control and our full objective differ as follows (Eq. (12)):

Fk−FSA=Ltaskk+LCPRk+LDGTk,Fk=Fk−FSA+LFSAk(θ,Pt).F_k^{-\mathrm{FSA}}=\mathcal L_{\mathrm{task}}^k+\mathcal L_{\mathrm{CPR}}^k+\mathcal L_{\mathrm{DGT}}^k,\qquad F_k=F_k^{-\mathrm{FSA}}+\mathcal L_{\mathrm{FSA}}^k(\theta,P^t).

Here PtP^t aggregates class summaries across clients and provides common semantic targets (Eq. (10)). The shared gate weights capacity, transfer and weak-pathway semantic penalties (Eq. (7)).

  • Under matched data, training and tuning budgets (C.3), our fusion/visual accuracy/rank on non-IID CREMA-D is 50.39/28.30/4.08, versus 46.91/23.42/2.87 for FedAvg+EBR (Table 6). On ModelNet40, fusion/weak accuracy is 84.20/65.95 versus 80.23/51.14. With the same shared-private basis, task-only training gives 43.17/20.53 fusion/visual accuracy on CREMA-D, versus our 50.39/28.30. Removing FSA while retaining CPR and DGT gives 48.50/27.08 (Table 16).

  • Recent work further clarifies the distinction. MFedPBA [1] aligns heterogeneous feature spaces and aggregates logit prototypes. ProMoE-FL [2] conditions feature synthesis on prototypes to recover missing modalities. Our focus is capacity loss in an available weak modality: CPR regularizes its private coordinates, while FSA anchors shared coordinates during federated training.

[1] Liao et al., Prototype-guided Bilateral Alignment Multimodal Federated Learning, ICML, 2026.

[2] Chhetri et al., ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities, MICCAI, 2026.

W2. Weak-pathway preservation and fusion performance

The tradeoff you identify is visible in our short-budget sweeps. At alpha = 0.5 in Table 13, our fusion accuracy is 28.87, below FedAvg (30.53) and FedAdam (34.42), while visual accuracy is higher at 21.76 versus 19.09 and 19.87. For these diagnostics, we used a single seed and mostly five rounds with one local epoch (C.11).

Under our main 100-round protocol, we obtain the highest mean in every non-IID setting of Table 1. On CREMA-D non-IID, our fusion accuracy is 50.39, compared with 49.72 for FedAdam and 45.39 for FedAvg. Our visual accuracy is 28.30 versus FedAvg's 22.09 (Table 6). On ModelNet40 IID, the corresponding fusion scores are 86.16 for our method and 86.18 for FedAdam.

W3. Shared/private dimension allocation

Our main experiments use an equal split. The allocation comparison keeps total width at 512, with the same initialization, data and training budget.

DatasetSharedPrivateFusion ↑Weak ↑Rank ↑
CREMA-D12838449.96 ± 1.0828.78 ± 1.364.31 ± 0.24
25625650.39 ± 0.9728.30 ± 1.414.08 ± 0.18
38412850.11 ± 1.1227.58 ± 1.533.76 ± 0.22
ModelNet4012838483.78 ± 0.7466.34 ± 1.177.94 ± 0.35
25625684.20 ± 0.6265.95 ± 1.047.58 ± 0.31
38412884.31 ± 0.6964.52 ± 1.267.09 ± 0.38

We report mean ± SD over three paired seeds under non-IID alpha = 0.5, following the evaluation protocol in Table 16. Fusion/Weak are accuracies (%), with Visual as Weak on CREMA-D and View-6 on ModelNet40. The 256/256 reference is from Table 6. Across allocations, we keep the dense basis and reconstruction size fixed, while the shared head and prototype sizes vary with shared width.

Questions

Thank you for these questions. We have discussed the contribution beyond EBR, the fusion/weak-pathway tradeoff and the dimension allocation alongside your corresponding comments, with the supporting evidence and comparison settings for each point.

AuthorsOpenReview ↗

To Dear Reviewer vtxE

Thank you for your careful review and patience. We have updated the rebuttal version of our manuscript, clarifying FedSBR’s contribution and adding the shared/private dimension-allocation study in Appendix C.13.

To help you locate the changes:

  • Light yellow: revisions addressing Reviewer vtxE.
  • Light blue: revisions addressing Reviewer 54zV.
  • Light purple: revisions addressing Reviewer Xaj3.

Updates to figures and tables are marked in their captions. For points raised by more than one reviewer, the color indicates the primary comment addressed.

Comments on the paper

1 comment

Dear reviewer vtxE and 54zV,

The authors have responded to your concerns. Please review their responses as soon as possible and provide any necessary feedback. Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.