acceptodds
Under review as a conference paper at ICLR 2027

Multimodal Federated Learning with Modality-Importance-Aware Compression

Abstract

Multimodal Federated Learning (MFL) has drawn significant attention as a decentralized training paradigm across multimodal edge devices. However, its communication cost, which grows linearly with the number of modalities, has emerged as a distinct scalability challenge. While prior approaches mitigate this through selective transmission of modality updates, they may sacrifice complementary information essential for multimodal learning. In this paper, we introduce a modality-importance-aware compression framework for MFL, termed Fed-MIAC, which retains multimodal benefits while improving communication efficiency. Specifically, Fed-MIAC differentiates compression rates across modality-specific model updates by jointly accounting for their relative importance and compression-induced distortion. Extensive simulations on diverse MFL benchmarks demonstrate that Fed-MIAC better preserves multimodal performance than importance-agnostic compression and modality selection baselines under constrained communication budgets, also validated by wall-clock overhead evaluations on embedded edge platforms. We further provide a convergence analysis that characterizes how modality-importance-aware compression affects the convergence rate.

Primary area:
optimization
Keywords:
Federated learning, Multimodal federated learning, Communication-efficient learning, Model compression, Modality importance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uHSZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Fed-MIAC, which aims to reduce the uplink communication overhead in multimodal federated learning. Clients first estimate modal importance using SHAPE scores based on Shapley values, then combine this with the singular values of model updates to allocate the compression level for each modality within a shared rank budget. Experimental results demonstrate the effectiveness of this method.

Strengths The paper is clearly written, and the logic behind the method is intuitive and easy to understand. The experiments are robust and demonstrate the method’s effectiveness.

Weaknesses

  1. The method is not particularly novel in essence; it primarily involves assessing modal importance and then allocating a budget to truncated SVD. Therefore, I would like the authors to explain what the paper’s greatest contribution is and which aspects represent new contributions.
  2. In the experimental section, I would like the authors to provide experimental results under the same communication costs (including both uplink and downlink) and to include metrics such as accuracy, client-side training time, and the number of communication rounds.
  3. Could the authors provide experiments under non-IID conditions? The current experiments are primarily conducted under IID conditions, but in real-world scenarios, non-IID conditions are more common.
Critical strengths

The paper is clearly written, and the logic behind the method is intuitive and easy to understand. The experiments are robust and demonstrate the method’s effectiveness.

Critical weaknesses
  1. The method is not particularly novel in essence; it primarily involves assessing modal importance and then allocating a budget to truncated SVD. Therefore, I would like the authors to explain what the paper’s greatest contribution is and which aspects represent new contributions.
  2. In the experimental section, I would like the authors to provide experimental results under the same communication costs (including both uplink and downlink) and to include metrics such as accuracy, client-side training time, and the number of communication rounds.
  3. Could the authors provide experiments under non-IID conditions? The current experiments are primarily conducted under IID conditions, but in real-world scenarios, non-IID conditions are more common.
Questions for authors
  1. The method is not particularly novel in essence; it primarily involves assessing modal importance and then allocating a budget to truncated SVD. Therefore, I would like the authors to explain what the paper’s greatest contribution is and which aspects represent new contributions.
  2. In the experimental section, I would like the authors to provide experimental results under the same communication costs (including both uplink and downlink) and to include metrics such as accuracy, client-side training time, and the number of communication rounds.
  3. Could the authors provide experiments under non-IID conditions? The current experiments are primarily conducted under IID conditions, but in real-world scenarios, non-IID conditions are more common.
6 comments
AuthorsOpenReview ↗

Response to Reviewer uHSZ - 1

We sincerely appreciate your feedback, which encouraged us to further explain the novelty of Fed-MIAC and broaden our experimental settings. We have provided detailed responses to Q1, Q2, and Q3 below and revised the main manuscript accordingly to address each question (Q2 in Appendix E.9 and Q3 in Appendix E.10; revisions are highlighted in blue).

Q1: Concern Regarding Novelty

The greatest contribution of this work is the development of an importance-aware compression framework for communication-efficient multimodal federated learning (MFL). Our framework addresses the limitations of directly applying importance-agnostic compression approaches developed for unimodal FL to MFL, where the heterogeneous importance of different modalities is overlooked. To operationalize the proposed framework, we formulate an importance-weighted distortion minimization problem that jointly accounts for modality importance and compression distortion under a shared communication budget.

Instantiating the proposed framework in practical MFL settings requires a modality importance measure and a compression scheme. We adopt SHAPE as a practical proxy for capturing the heterogeneous importance of different modalities, while truncated SVD is employed because its rank-dependent compression distortion can be explicitly characterized, enabling tractable optimization of rank allocation. We consider this combination well suited to illustrating and evaluating the proposed importance-aware resource allocation principle. Nevertheless, we agree with the reviewer that the novelty of our work does not lie in these individual methods or their straightforward combination. Instead, their use represents a concrete instantiation of the proposed framework, which is not inherently restricted to either technique. In Section 6.2 of the revised manuscript (previously Appendix E.2 - Applying modality importance to quantization-based compression), we also extend the importance-aware resource allocation principle to quantization-based compression and observe performance improvements, providing further evidence that the effectiveness of the proposed approach is not limited to SVD-based compression.

Finally, we empirically validate the proposed framework across diverse MFL settings and communication budgets. In addition to comparisons with existing modality-selection frameworks, we evaluate uniform rank allocation and importance-agnostic rank allocation to examine the specific benefits of incorporating modality importance into compression resource allocation. The observed performance improvements indicate that the benefits of the proposed method are not solely attributable to SVD-based compression, but also to the importance-aware allocation of communication resources. These results provide empirical support for the effectiveness of the proposed optimization framework in communication-constrained MFL.

AuthorsOpenReview ↗

Response to Reviewer uHSZ - 2

Q2: Additional experiments regarding communication cost

We evaluate communication efficiency using two measures: the number of communication rounds and the cumulative communication cost (in gigabits), including both uplink and downlink transmissions, required to reach a target F1 score or accuracy.

Fed-MIAC requires fewer rounds and lower cumulative communication cost to achieve the same performance as the baselines.

As shown in Table 2 of the main manuscript, the per-round training times are 0.88, 0.66, and 2.97 seconds for CrisisMMD, KU-HAR, and PTB-XL, respectively. Since the compared methods follow the same local training procedure, we report the number of communication rounds as a measure of training efficiency.

F1 vs Round

DatasetCrisisMMDKU-HARPTB-XL
Target F1304040504050
SVDFed38—10918071116
MCB35150871274181
MFedMC40191791384186
fedMIAC35134771023671

F1 vs Comm. Cost

DatasetCrisisMMDKU-HARPTB-XL
Target F1304040504050
SVDFed10.63—14.7024.2611.0117.95
MCB8.7037.2810.8015.775.8211.51
MFedMC9.9447.479.8117.145.8212.22
fedMIAC8.7033.339.5712.685.1210.11

Acc vs Round

DatasetCrisisMMDKU-HARPTB-XL
Target Acc (%)657060704550
SVDFed3593150—56131
MCB3564148—26111
MFedMC3761138—3691
fedMIAC34501291992161

Acc vs Comm. Cost

DatasetCrisisMMDKU-HARPTB-XL
Target Acc (%)657060704550
SVDFed9.7325.9220.21—8.6920.26
MCB8.7015.9018.38—3.6915.78
MFedMC9.1915.1617.14—5.1112.93
fedMIAC8.4512.4316.0424.742.998.68
AuthorsOpenReview ↗

Response to Reviewer uHSZ - 3

Q3: Additional experiments on non-IID conditions

We constructed the datasets following the existing benchmark. Specifically, CrisisMMD was partitioned in an IID manner, while PTB-XL, KU-HAR, and CMU-MOSEI were partitioned using natural splits based on Site, Person ID, and Video Index, respectively.

These ID- or index-based natural splits reflect the inherent heterogeneity of the datasets and therefore should not be assumed to be IID. Artificially repartitioning the data using methods such as a Dirichlet distribution could distort their inherent distributional characteristics. Therefore, we preserved the natural splits of the existing benchmark.

We also conducted experiments using non-IID partitions of CrisisMMD generated from a Dirichlet distribution with α∈1,0.5\alpha \in \\{1, 0.5 \\}, where α\alpha is the concentration parameter and smaller values indicate greater data heterogeneity across clients. The results show that Fed-MIAC consistently achieves the best performance across all evaluated rank configurations under both non-IID settings.

α=1\alpha=1

AlgorithmF1Rank 32Rank 16Rank 8Rank 4
FedAvg43.95————
SVDFed33.41————
MCB—34.1835.0833.4433.60
MFedMC—36.0835.3934.5333.65
fedMIAC—44.5640.8240.4635.05

α=0.5\alpha=0.5

AlgorithmF1Rank 32Rank 16Rank 8Rank 4
FedAvg33.62————
SVDFed25.95————
MCB—32.4030.8627.8329.16
MFedMC—32.1129.8431.2831.92
fedMIAC—33.5433.7433.6433.49
Reviewer_cPYUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Fed-MIAC for communication-efficient multimodal federated learning. It combines locally estimated SHAPE modality importance scores with the singular values of model updates to allocate SVD ranks under a shared rank budget. Experiments on four multimodal benchmarks report improvements over modality selection and compression baselines. The paper also evaluates computation and communication costs on Jetson TX2 and provides a nonconvex analysis characterizing the effects of compression residuals.

Critical strengths
  1. The formulation jointly considers modality importance and update reconstruction error. Proposition 4.1 provides an exact solution to the stated rank allocation problem, and the comparisons with Uniform and w/o MIAC in Table 3 support the empirical value of incorporating modality importance.

  2. The evaluation covers multiple datasets, communication budgets, and two underlying FL algorithms. The Jetson TX2 evaluation additionally accounts for the computation required by SHAPE estimation and SVD, providing useful evidence about the practical trade-off.

Critical weaknesses
  1. The novelty is limited. The main contribution combines an existing modality importance estimator with SVD compression. The allocation rule in Proposition 4.1 follows directly from sorting the weighted marginal gains of the separable objective in Eq. (4). While the combination is useful, the conceptual advance beyond these established components appears incremental.

  2. The theory provides limited support for the specific importance-aware allocation. The bound in Eq. (73) contains a compression-dependent term that is not guaranteed to vanish, as acknowledged in Remark 3. Its implications under the evaluated fixed rank budgets therefore remain limited. Moreover, the distortion quantity in Eq. (74) uses gradient-based modality weights, whereas the algorithm uses SHAPE-based weights. The analysis does not establish a quantitative relationship between them, so the theoretical benefit of the proposed allocation remains unclear. The full-participation assumption also limits direct applicability to the client-sampling setting used in the experiments.

Questions for authors
  1. Remark 3 explains why truncated SVD minimizes reconstruction error for a given modality-wise rank allocation. Could the authors clarify what quantitative relationship, if any, the current analysis establishes between the SHAPE-weighted objective in Eq. (4) and the gradient-weighted distortion in Eq. (74)?

  2. Theorem 5.1 assumes full participation, while Appendix D.3 uses partial client participation. Could the authors clarify how client sampling affects the bound and the applicability of the theoretical conclusions to the experimental setting?

3 comments
AuthorsOpenReview ↗

Response to Reviewer cPYU - 1

We sincerely appreciate your thoughtful and constructive feedback, which encouraged us to further clarify the novelty of Fed-MIAC and strengthen the connection between our theoretical analysis and empirical results, particularly regarding modality importance weights and partial client participation. We agree with the reviewer that establishing a clear connection between the theoretical analysis and experimental results is important for understanding the theoretical motivation behind the empirical performance improvements achieved by Fed-MIAC

We provide detailed responses to W1, W2, Q1, and Q2 below and revised the main manuscript accordingly to address each question (Q1 in Section 6.2 and Q2 in Appendix C.9; revisions are highlighted in blue).

W1: Concern Regarding Novelty

We agree with the reviewer that SHAPE and truncated SVD are established techniques and that the allocation rule in Proposition 4.1 follows directly from sorting the weighted marginal gains of the separable objective in Eq. (4). The proposition is not intended to introduce a new optimization principle, but rather to provide an exact and efficient solution to the formulated problem.

The primary contribution of our work lies in developing an importance-aware compression framework for communication-efficient multimodal federated learning (MFL). To operationalize this framework, we formulate an importance-weighted distortion minimization problem that jointly accounts for modality importance and compression distortion under a shared communication budget. In this context, SHAPE and truncated SVD serve as a concrete instantiation of the proposed framework rather than constituting its primary source of novelty. We consider this combination suitable for implementing and evaluating the proposed design principle, as SHAPE provides an intuitive and practical proxy for modality importance and SVD enables tractable distortion-based rank allocation.

Furthermore, Section 6.2 of the revised manuscript (previously Appendix E.2: Applying modality importance to quantization-based compression) demonstrates performance improvements when the proposed framework is applied to a quantization-based compression method, providing further evidence of its applicability beyond SVD. Thus, the conceptual contribution of our work lies in formulating importance-aware compression tailored to MFL and validating its effectiveness, rather than in the novelty of the individual techniques or their combination.

W2: Alignment between theory and experiments

We would like to clarify that our convergence analysis is not intended to guarantee that the compression-dependent term vanishes. Specifically, even with exact knowledge of the gradient-based modality weights, a finite rank budget does not guarantee that DgaprD_{\mathrm{gap}}^r converges to zero. Rather, our objective is to mitigate this potential convergence gap under a limited communication budget.

To further clarify the theoretical implications, we quantitatively examine the relationship between gradient-based and SHAPE-based modality weights in our response to Q1 and further clarify how DgaprD_{\mathrm{gap}}^r changes under partial client participation in our response to Q2.

AuthorsOpenReview ↗

Response to Reviewer cPYU - 2

Q1: Quantitative relationship between the SHAPE-weight and the gradient-based weight.

\sum_{m=1}^M \alpha_m D(W_m; \Lambda_m) \quad\text{s.t.} \quad \sum_{m=1}^M \Lambda_m \le \Lambda, \Lambda_m \in \mathbb{Z}_{\ge 0}.$$ $$Eq. (74):\qquad \mathcal D_{\mathrm{gap}}^r \triangleq{} \mathbb E [ G^2 \times \sum_{i=1}^{N} \sum_{m=1}^{M} p_i \frac{||\nabla_{w_m} F(w^r)|| _F}{G} ||e_{i,m}^r||_F^2 ].$$ To quantitatively examine the relationship between $\alpha_m$ in Eq. (4) and $\frac{\left\|\nabla_{w_m}F(w^r)\right\|_F}{G}$ in Eq. (74), we provide a quantitative comparison of the SHAPE-based and global gradient-based importance weights across communication rounds in the table below. We note that the importance-agnostic variant (w/o MIAC) uses fixed uniform weights of (0.5, 0.5) for the two modalities. The results reveal that the SHAPE-based weights generally align with the gradient-based weights. This empirical agreement supports the use of SHAPE as a proxy for the theoretical weights, as shown in the table below (also reported in **Table 5 of the main manuscript**). | Dataset | CrisisMMD | | | | KU-HAR | | | | PTB-XL | | | | | -------------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | | Round | 50 | 100 | 150 | 200 | 50 | 100 | 150 | 200 | 50 | 100 | 150 | 200 | | Gradient-based ratio | 0.62:0.38 | 0.63:0.37 | 0.69:0.31 | 0.68:0.32 | 0.84:0.16 | 0.82:0.18 | 0.77:0.23 | 0.73:0.27 | 0.43:0.57 | 0.46:0.54 | 0.43:0.57 | 0.52:0.48 | | SHAPE-based ratio | 0.72:0.28 | 0.73:0.27 | 0.76:0.24 | 0.69:0.31 | 0.80:0.20 | 0.65:0.35 | 0.66:0.34 | 0.78:0.22 | 0.49:0.51 | 0.49:0.51 | 0.49:0.51 | 0.49:0.51 | # Q2: Convergence of partial client participation **In the revised manuscript, we have also updated Theorem 5.1 to present a generalized form covering both full and partial client participation.** We extend the convergence analysis to partial client participation, where $\mathcal S^r\subseteq[N]$ denotes the set of participating clients at round $r$, with $|\mathcal S^r|=S$, as follows: $$\frac{1}{R}\sum_{r=0}^{R-1}\mathbb E\|\nabla F(w^r)\|^2 \leq \mathcal B_{\mathrm{FL},S} +\frac{4}{TR}\sum_{r=0}^{R-1}\bigl(\mathcal D_{\mathrm{gap},S}^r\bigr)^{1/2} +\frac{L\eta}{TR}\sum_{r=0}^{R-1} \left(4\mathcal M_{\mathrm{dir},S}^r+2\mathcal E_{\mathrm{agg},S}^r\right).$$ Under Assumptions (C.1–C.5) in the main manuscript, $\mathcal B_{\mathrm{FL},S}$ captures standard federated optimization errors, including an additional component arising from client sampling, separately from compression effects. For the compression-related terms, the contributions of $\mathcal M_{\mathrm{dir},S}^r$ and $\mathcal E_{\mathrm{agg},S}^r$ to the convergence bound decrease at a rate of $O(R^{-1/2})$, under the learning-rate choice $\eta=O(R^{-1/2})$ and boundedness of these terms. In contrast, the contribution of the gradient-weighted compression distortion term $\mathcal D_{\mathrm{gap},S}^r$ is not guaranteed to vanish. Specifically, this term is expressed as $$\mathcal D_{\mathrm{gap},S}^r =\mathbb E\Biggl[ G^2\frac{1}{S}\sum_{i\in\mathcal S^r}\sum_{m=1}^M \frac{||\nabla_{w_m}F(w^r)||_F}{G}||e_{i,m}^r||_F^2 \Biggr].$$ Importantly, this term retains the same gradient-weighted compression distortion structure as in the full-participation case, with the full-client weighted sum replaced by an average over participating clients. Therefore, despite the additional effects of client sampling, the theoretical motivation for Fed-MIAC, which prioritizes reducing compression errors in modalities with higher loss sensitivity, remains applicable under partial client participation.
AuthorsOpenReview ↗

Kind request for a response to our rebuttal

As the discussion period draws to a close, we would be grateful if the reviewers could review our responses and the revised manuscript and share any remaining questions or concerns. We would be happy to address them before the discussion closes.

Reviewer_m7g8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to reduce uplink communication in multimodal federated learning. It proposes Fed-MIAC, which uses a local Shapley-based score to estimate the importance of each modality. It then assigns different truncated-SVD ranks to modality updates under a shared rank budget. The paper also gives a convergence bound for compressed federated learning. Experiments on four multimodal datasets show better F1 scores than the reported modality-selection and compression baselines in several settings. The paper further reports client-side measurements on an edge device.

Critical strengths
  • Fed-MIAC can send compressed updates from several modalities instead of choosing only one.
  • The paper derives a simple rule for allocating ranks using both modality importance and singular values. Its convergence analysis also makes the effect of compression error explicit.
Critical weaknesses
  • A key term in the convergence bound weights compression errors using global gradient norms, while Fed-MIAC allocates ranks using locally computed Shapley-based scores. The paper does not establish how closely these two quantities agree.
Questions for authors
  • How are the local Shapley-based scores related to the global gradient weights in the convergence bound? Are there assumptions under which they rank modalities in a similar way?
  • Does the convergence bound support the proposed rank allocation rule specifically? If the two sets of weights disagree, does the bound still predict an advantage over uniform rank allocation?
4 comments
AuthorsOpenReview ↗

Response to Reviewer m7g8 - 1

We sincerely appreciate your insightful and constructive feedback, which encouraged us to further clarify the relationship between global gradient-based weights and local SHAPE-based weights. We agree with the reviewer that further characterizing the alignment between these weights is important for understanding how the theoretical analysis in Theorem 5.1 relates to the empirical performance improvements achieved by Fed-MIAC.

To address this point, we have incorporated the theoretical analyses discussed in Q1 into Appendix F and the numerical results presented in Q2, along with their accompanying discussion, into Section 6.2 of the revised manuscript. These additions are highlighted in blue, and we provide detailed responses to Q1 and Q2 below.

Before the theoretical analyses, we introduce the following key notation.

  • Fi(S,w)F_i(S,w) denotes the loss of client ii evaluated at model ww, with the inputs of modalities outside SS set to zero. We write Fi(w)=Fi(M,w)F_i(w)=F_i(\mathcal M,w) for the loss with all modality inputs.
  • ϕi,m(w)\phi_{i,m}(w) denotes the input-space Shapley value of modality mm for client ii, using the negative loss −Fi(S,w)-F_i(S,w) as the utility.
  • w−mw^{-m} denotes the model obtained by zeroing wmw_m, the encoder parameter block associated with modality mm, while keeping all other parameters fixed. The resulting loss increase, with all modality inputs retained, is di,m(w)=Fi(M,w−m)−Fi(M,w).d_{i,m}(w) = F_i(\mathcal M,w^{-m})-F_i(\mathcal M,w).
AuthorsOpenReview ↗

Response to Reviewer m7g8 - 2

Q1: Relationship between local Shapley-based scores and global gradient weights

We address Q1 from two theoretical perspectives, introducing four additional assumptions and two propositions for analysis. These assumptions and propositions are labeled F.1–F.6 to reflect their placement in Appendix F.

1. Connecting Local Shapley Values to Global Gradient Norms

Assumption F.1 connects the input-space Shapley scores to the loss change induced by zeroing the corresponding modality-specific encoder parameters. A first-order Taylor approximation relates this loss change to the gradient inner product, and Assumption F.2 relates that inner product to the gradient norm.

Assumption F.1 (Shapley approximation by parameter zeroing). At the shared model ww, the difference between the input-space Shapley value and the loss increase from parameter zeroing is bounded by ϵd≥0\epsilon_d\geq 0 for all clients i∈[N]i\in[N] and modalities m∈[M]m\in[M], i.e., ∣ϕi,m(w)−di,m(w)∣≤ϵd.|\phi_{i,m}(w)-d_{i,m}(w)|\leq\epsilon_d.

Assumption F.2 (Positive alignment). For the modality-specific encoder parameter blocks and their corresponding local gradients, there exists a constant c∈(0,1]c\in(0,1] for all clients i∈[N]i\in[N] and modalities m∈[M]m\in[M], i.e.,

\le \left\langle \nabla_m F_i(w),-w_m\right\rangle_F.$$ **Proposition F.3 (Shapley-gradient relationship).** Under Assumptions F.1 and F.2, let $G_m:=||\nabla_m F(w)||_F$ denote the global gradient norm of modality $m$, where $F(w)=\sum_{i=1}^{N}p_i F_i(w)$ and $\sum_{i=1}^{N}p_i=1$ with $p_i\geq 0$. For $||w_m||_F>0$, the following bound holds: $$G_m \lesssim \frac{\sum_{i=1}^{N}p_i\phi_{i,m}(w)+\epsilon_d} {c||w_m||_F}.$$ **(Remark)** The global gradient norm can be approximately upper-bounded in terms of the aggregated local Shapley-based scores, the encoder parameter norm, and the gradient alignment constant $c$. Although this does not establish direct proportionality between the two importance measures, it provides a theoretical connection between Shapley-based modality importance and gradient-based modality weights. ## **2. Conditions for similar modality rankings** We further investigate the conditions under which local Shapley-based modality rankings agree with global gradient-weight rankings in the two-modality case, aiming to provide theoretical insight into their relationship. Building on Assumptions C.1 and C.3 (L-smoothness and client heterogeneity) in Appendix C of the main manuscript, we introduce two additional assumptions, F.4 and F.5. Assumption F.4 specifies a sufficient global gradient gap to preserve gradient rankings at the client level, while Assumption F.5 provides encoder-norm and gradient-alignment conditions to relate local gradient rankings to Shapley-based modality rankings. **Assumption F.4 (sufficient global gradient gap).** The difference between the global gradient norms of two modality encoders is larger than $2\delta_i$ for every client $i\in[N]$, i.e., $$G_1-G_2>2\delta_i,$$ where $\delta_i\triangleq L\|\Delta_i\|+\sigma_g$, $\Delta_i$ is the local model update, $L$ is the smoothness constant (Assumption C.1), and $\sigma_g$ is an upper bound for heterogeneity (Assumption C.3). **Assumption F.5 (Equal encoder norms and approximate gradient alignment).** The two modality-specific encoder blocks have the same Frobenius norm, and every nonzero local block gradient is approximately aligned with its encoder-zeroing direction with a constant $\epsilon_\theta\in[0,1)$ for every client $i\in[N]$, i.e., $$||w_{i,1}||_F=||w_{i,2}||_F>0, \quad \cos\theta_{i,m}(w_i)\ge1-\epsilon_\theta,$$ where $\theta_{i,m}(w_i)$ denotes the angle between $\nabla_m F_i(w_i)$ and $-w_{i,m}$. **Proposition F.6 (Approximate modality-ranking consistency).** Under Assumptions F.1-F.2 and F.4-F.5, and using a first-order Taylor approximation, a sufficient condition for the local Shapley-based modality ranking to agree with the global gradient-based ranking ($G_1>G_2$) is $$\frac{G_1-G_2}{G_1}> (2+\epsilon_\theta)\frac{\delta_i}{G_1} +\epsilon_\theta +\frac{2\epsilon_d}{AG_1},$$ where $A=||w_{i,1}||_F=||w_{i,2}||_F>0$. **(Remark)** This condition provides theoretical insight into when local SHAPE-based importance can reflect global gradient-based modality rankings. Specifically, when the relative global importance gap sufficiently exceeds the combined effects of client heterogeneity, local drift, gradient misalignment, and Shapley approximation error, the two rankings are expected to agree.
AuthorsOpenReview ↗

Response to Reviewer m7g8 - 3

Q2: Relation of the rank allocation rule between the convergence bound and Fed-MIAC

We address Q2 with experimental results and have revised Section 6.2, Experiments, of the manuscript to include the following results.

:= \mathbb E \left[ G^2 \sum_{i=1}^{N} \sum_{m=1}^{M} p_i \cdot \frac{||\nabla_{w_m} F(w^r)||_F}{G} \cdot ||e_{i,m}^r||_F^2 \right]$$ For the $\mathcal D_{\mathrm{gap}}^r$ in the convergence bound, the normalized global gradient norms $(\frac{\|\nabla_{w_m}F(w^r)\|_F}{G})$ serve as modality-wise weights for compression errors, reflecting relative modality-wise loss sensitivity. In contrast, Fed-MIAC relies on locally estimated SHAPE weights for rank allocation. To examine whether the convergence analysis supports the proposed rank allocation, we quantitatively assess the agreement between these two sets of weights. Specifically, we compare their normalized modality-wise weights across communication rounds, as shown in the table below (also reported **in Table 5 of the main manuscript**). | Dataset | CrisisMMD | | | | KU-HAR | | | | PTB-XL | | | | | -------------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | | Round | 50 | 100 | 150 | 200 | 50 | 100 | 150 | 200 | 50 | 100 | 150 | 200 | | Gradient ratio | 0.62:0.38 | 0.63:0.37 | 0.69:0.31 | 0.68:0.32 | 0.84:0.16 | 0.82:0.18 | 0.77:0.23 | 0.73:0.27 | 0.43:0.57 | 0.46:0.54 | 0.43:0.57 | 0.52:0.48 | | Shapley ratio | 0.72:0.28 | 0.73:0.27 | 0.76:0.24 | 0.69:0.31 | 0.80:0.20 | 0.65:0.35 | 0.66:0.34 | 0.78:0.22 | 0.49:0.51 | 0.49:0.51 | 0.49:0.51 | 0.49:0.51 | We clarify that the importance-agnostic variant (w/o MIAC) assigns fixed uniform importance weights of (0.5, 0.5) to the two modalities. The results show that SHAPE generally captures the relative modality importance reflected by the global gradient-based weights without requiring access to the global objective. In particular, SHAPE reflects the pronounced modality imbalance in KU-HAR and the relatively balanced importance in PTB-XL. These observations, together with the results in Table 4 of the original manuscript, suggest that larger discrepancies between uniform importance weights and global gradient-based weights can be associated with larger performance gaps between Fed-MIAC and w/o MIAC. In particular, KU-HAR exhibits the most pronounced modality imbalance and the largest performance gap between the two methods. Notably, w/o MIAC even underperforms uniform rank allocation on KU-HAR, indicating that minimizing compression distortion with mismatched importance weights does not necessarily outperform uniform rank allocation when modality sensitivities are highly imbalanced.
AuthorsOpenReview ↗

Kind request for a response to our rebuttal

As the discussion period draws to a close, we would be grateful if the reviewers could review our responses and the revised manuscript and share any remaining questions or concerns. We would be happy to address them before the discussion closes.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions and Responses

Dear Reviewers,

Thank you for your time and effort. Below, we summarize the major revisions to our manuscript and key clarifications provided in the rebuttal. All changes are highlighted in blue\color{blue}{\mathrm{blue}} in the revised manuscript.

1. Relationship between global gradient-based weights and local SHAPE-based weights (Reviewers m7g8 and cPYU)

  • Added quantitative comparison of global gradient-based weights and local SHAPE-based weights in Section 6.2 (Table 5).
  • Added theoretical analysis connecting global gradient norms and local SHAPE-based importance, including a condition for preserving modality ranking in Appendix F.
  • These results clarify the relationship between the local SHAPE-based weights used in Fed-MIAC and the global gradient-based weights used in our convergence analysis.

2. Convergence analysis under partial client participation (Reviewer cPYU)

  • Generalized the convergence analysis in Section 5.1 (Theorem 5.1) to cover both full and partial client participation, and provided a detailed proof in Appendix C.9.
  • This extension supports the theoretical motivation for MIAC under partial participation, consistent with the experimental setting.

3. Additional experiments (Reviewer uHSZ)

  • Added experiments comparing communication costs required to reach the same target performance in Appendix E.9 (Tables 12-15).
  • Added experiments under non-IID data distributions generated using Dirichlet distributions with α=0.5\alpha=0.5 in Section 6.2 (Table 3) and α=1\alpha=1 in Appendix E.10 (Table 16).
  • These results further support Fed-MIAC's communication efficiency and effectiveness under heterogeneous data distributions.

4. Novelty of Fed-MIAC (Reviewers cPYU and uHSZ)

  • Clarified that the main contribution of Fed-MIAC lies in developing an importance-aware compression framework for MFL that accounts for both modality importance and compression distortion, without being inherently limited to SHAPE or SVD.
  • Demonstrated that Fed-MIAC outperforms the evaluated importance-agnostic and modality-selection methods, while extending the proposed framework beyond SVD to quantization-based compression.

We sincerely appreciate the reviewers' consideration of our revisions.

AuthorsOpenReview ↗

Reminder: Rebuttal Discussion Period Ending Soon

Dear Reviewers,

With about 24 hours remaining in the rebuttal discussion period, please let us know if any of your concerns remain unaddressed or need further clarification.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.