Operator-Space Federated Learning with Diffusion-Based Neural Operators for Heterogeneous Multimodal Sensing

Federated multimodal human sensing involves substantial heterogeneity across sensor modalities, spatial and temporal resolutions, local tokenizations, and client data distributions. We formulate federation over a shared semantic operator representation that supports aggregation across these heterogeneous local discretizations. A Spectral-Gated Linear Transformer constructs the semantic matrix $M=\varphi (K{)}^{\top }V$, whose dimension is independent of the local token count, and applies spectral gating to retain dominant shared structure while suppressing modality-specific residual components. We further introduce Diffusion-Based Neural Operator Aggregation for server-side federation. Client semantic updates are represented in low-dimensional spectral coordinates, where a diffusion model learns the distribution used to form the global semantic update. The remaining model parameters are aggregated with FedAvg. A Subspace-Projected Differential Privacy mechanism protects the released semantic updates using a projection basis derived from the previously broadcast global semantic operator. Under the stated clipping and noise assumptions, each privatized semantic-update release satisfies $(ϵ,\delta )$-differential privacy. Experiments on MM-Fi evaluate pose estimation, action recognition, and localization under heterogeneous client discretizations, non-IID data, missing modalities, and matched per-release privacy settings. The proposed operator-space formulation consistently improves performance over federated baselines across these settings. Additional evaluation on RELI11D demonstrates zero-adaptation cross-dataset out-of-distribution transfer with partially overlapping sensor stacks. Ablations further show complementary contributions from the local spectral representation and the learned server-side aggregation, with diffusion-based aggregation improving over simple averaging in the same spectral coordinates.