acceptodds
Under review as a conference paper at ICLR 2027

Operator-Space Federated Learning with Diffusion-Based Neural Operators for Heterogeneous Multimodal Sensing

Abstract

Federated multimodal human sensing involves substantial heterogeneity across sensor modalities, spatial and temporal resolutions, local tokenizations, and client data distributions. We formulate federation over a shared semantic operator representation that supports aggregation across these heterogeneous local discretizations. A Spectral-Gated Linear Transformer constructs the semantic matrix , whose dimension is independent of the local token count, and applies spectral gating to retain dominant shared structure while suppressing modality-specific residual components. We further introduce Diffusion-Based Neural Operator Aggregation for server-side federation. Client semantic updates are represented in low-dimensional spectral coordinates, where a diffusion model learns the distribution used to form the global semantic update. The remaining model parameters are aggregated with FedAvg. A Subspace-Projected Differential Privacy mechanism protects the released semantic updates using a projection basis derived from the previously broadcast global semantic operator. Under the stated clipping and noise assumptions, each privatized semantic-update release satisfies -differential privacy. Experiments on MM-Fi evaluate pose estimation, action recognition, and localization under heterogeneous client discretizations, non-IID data, missing modalities, and matched per-release privacy settings. The proposed operator-space formulation consistently improves performance over federated baselines across these settings. Additional evaluation on RELI11D demonstrates zero-adaptation cross-dataset out-of-distribution transfer with partially overlapping sensor stacks. Ablations further show complementary contributions from the local spectral representation and the learned server-side aggregation, with diffusion-based aggregation improving over simple averaging in the same spectral coordinates.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
federated learning, multimodal representation learning, neural operators, heterogeneous discretizations, spectral learning, diffusion models, differential privacy, human sensing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_S43zOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a federated learning framework for multimodal human sensing with heterogeneous sensors. It has three parts. (1) SGLT is a linear-attention encoder that applies a soft singular-value gate to the matrix M = φ(K)ᵀV. (2) Diff-NOA projects client updates of this "semantic block" onto the top-r singular directions of the global block, trains a small score model on the resulting r²-dimensional vectors each round, and uses a reverse-diffusion sample as the server update; all other parameters use FedAvg. (3) SP-DP clips the semantic update and adds Gaussian noise with a larger scale outside the top-r subspace. Experiments on MM-Fi report pose, action and localization results against many federated baselines, with a transfer test to RELI11D and several robustness sweeps.

Critical strengths

Relevant problem. Federated training across clients with different sensor sets, resolutions and missing modalities, under privacy constraints, is a real need in device-free sensing, and the paper tries to evaluate all of these stressors in one setting.

Scope of the privacy claim is stated. The paper says the guarantee is per release and covers only the semantic update, which at least makes the limits of the claim visible.

Critical weaknesses

[The method is not well defined, and its three components are not justified]

  • What is aggregated?: M = φ(K)ᵀV is an activation that changes with every input, yet the paper treats θ_M as a parameter block with client updates ΔM_k and a server-side SVD. How an input-dependent matrix becomes a federated parameter is never explained. The stated motivation, that clients with different token counts cannot be aggregated, also does not hold: transformer weights do not depend on sequence length, so FedAvg already applies. The "discretization invariance" lemma only states that a d×d matrix is d×d.

  • Why diffusion?: Aggregation needs a point estimate, but Diff-NOA draws one sample from a model of the client-update distribution. The score model is trained on about 40 vectors of dimension 256 per round, a regime where it can do little more than memorize them, so the sample is close to one client's update. No argument is given for why a sample should beat the mean, a median or a trimmed mean. The interaction with DP makes this harder to accept: by my rough calculation from Eq. (9), at ε = 2 and C = 1 the noise has σ ≈ 2.4 per coordinate, so after projection the noise norm is on the order of 40 while the clipped signal norm is at most 1. Averaging over clients is what reduces such noise; sampling does not. Yet the paper reports that sampling outperforms averaging in the same coordinates (Table 9) and that privacy costs only 1.9 mm in MPJPE (Table 1).

  • Neural operator: Nothing in the method learns a map between function spaces. The term refers to a 2-layer MLP score network over a 256-dimensional vector.

  • SP-DP: Theorem B.4 shows the mechanism is isotropic Gaussian noise at scale σ_sig plus extra noise in the complementary subspace. It is therefore never less noisy than the standard Gaussian mechanism at the same ε, and any utility gain must come from the server's projection, which is post-processing available to the isotropic baseline too. As I read Eq. (10), that projection removes the extra noise exactly, so κ should have no effect on the aggregated update, yet Table 7 reports large effects of κ. The text is also inconsistent about where noise goes ("confine noise to the signal's null space" vs. noise in both subspaces). The calibration in Eq. (9) is the classical bound, which is stated for ε < 1 (see Balle & Wang, ICML 2018) but is used up to ε = 8.

  • Privacy scope: Only ΔM is privatized; all other parameter updates are uploaded in the clear, and the guarantee is per round over hundreds of rounds with no composition. Comparing "at matched ε" with DP-FedAvg, which perturbs the whole model, is not a like-for-like comparison, and the gradient-inversion test attacks only the protected block.

[The empirical evidence is internally inconsistent and the positioning is weak]

  • Inconsistent numbers: In Table 1 the "LAT (no gate)" baseline and the "w/o SGLT (LAT)" ablation have identical means on all three tasks (89.8 / 87.4 / 12.2) with different standard deviations, although one is a plain federated baseline and the other should include Diff-NOA and SP-DP. Table 3 reports 360 rounds to target at q = 0.4, while Table 8 in the appendix reports 160 rounds at q = 0.4 and 360 at q = 0.1. The main text uses T = 1000 rounds and the appendix T = 200.

  • Missing or undefined baselines: CA2FL, HPFL and Decoupled Subgraph FL are listed as baselines but appear in no table. FedRod, FedGCN and FedGAT appear in tables without description. It is unclear how graph-based FL methods apply to this task.

  • Protocol gaps: How 100 clients are formed from MM-Fi, which modalities each client type holds, how the modality-specific encoders are shared, and how localization is evaluated on RELI11D without adaptation are not specified.

  • Related work: Diffusion-based server aggregation exists (pFedGPA, Lai et al., AAAI 2025). Projecting private gradients onto a public low-dimensional subspace before or after adding noise is established (Zhou et al., ICLR 2021; GEP, Yu et al., ICLR 2021). Multimodal FL with heterogeneous or missing modalities has dedicated methods and benchmarks (Harmony, Ouyang et al., MobiSys 2023; CreamFL, Yu et al., ICLR 2023; FedMultimodal, Feng et al., KDD 2023). None are cited or compared, while the references given for "generative FL" include a paper on DP convergence. The comparison is instead filled with generic FL optimizers.

Questions for authors

What exactly is θ_M? Which trainable parameters does it contain, and how is a client "update" ΔM_k of an input-dependent matrix computed?

Why is a single diffusion sample a better server update than the mean, coordinate-wise median or trimmed mean of the same projected vectors? Please report these baselines under identical DP noise, and the variance of the server update across rounds.

Is the score model re-initialized each round or carried over? With about 40 training vectors, how do its samples differ from the training points?

What is the signal-to-noise ratio of the projected update at ε = 2? How can the utility loss from privacy be this small?

Given the projection in Eq. (10), through which path does κ change the result in Table 7?

Are the non-semantic parameter updates privatized in any experiment? If not, what does "matched ε" mean relative to DP-FedAvg?

Please explain the identical rows in Table 1 and the discrepancy between Table 3 and Table 8, and report the listed baselines that are missing from the tables.

How are clients, modalities and encoders assigned, and how is RELI11D localization evaluated?

Minor: several references to papers from 2020 onward give the venue as "NIPS" ([24], [32], [42], [46]); these should read NeurIPS. The manuscript header ("ICLR 2026"), the unresolved references ("Fig. ??", "Sec. ??") and the duplicated paragraphs in Sections 4.1 and 4.2 should also be fixed.

Reviewer_9oy8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes an operator-space federated learning framework for heterogeneous multimodal sensing. In a nutshell, SGLT builds a fixed-dimensional semantic matrix M=ϕ(K)⊤VM=\phi(K)^⊤V, Diff-NOA aggregates the clients' semantic updates in a low-dimensional spectral space using diffusion modeling, and SP-DP adds subspace-aware noise to handle privacy. On the experimental side, the authors evaluate on MM-Fi across pose estimation, action recognition, and localization, covering discretization heterogeneity, non-IID data, missing modalities, and privacy constraints. They also look at cross-dataset transfer to RELI11D and include fairly extensive ablations and scalability analyses.

Critical strengths
  1. Experiments are thorough, with three tasks, a solid set of FL baselines, heterogeneity stress tests, privacy experiments, OOD transfer, communication efficiency, component ablations, and scalability studies.

  2. The problem setting is meaningful, since handling modality and discretization heterogeneity, non-IID data, missing modalities, and privacy all at once is a hard and practically relevant combination for federated sensing.

Critical weaknesses
  1. I'm not fully convinced the core claims about the operator space and diffusion aggregation are justified. The fact that M=ϕ(K)⊤VM=ϕ(K)⊤^V has a fixed d×d size does show dimensional compatibility across different token counts, but that alone doesn't establish semantic alignment across modalities. I also don't quite see why modality-specific noise should mainly sit in the trailing singular directions, or why a random sample from the learned client-update distribution would count as a meaningful "consensus" update for the federated objective.

  2. The differential-privacy guarantee also looks incomplete, and possibly miscalibrated. The proof assumes that clipping an update to norm C gives sensitivity C, but this needs clarifying—under a standard replace-one adjacency, the difference between two clipped updates can actually reach 2C. On top of that, SP-DP only protects ΔMkΔM_k, while Δθ_{\M} is transmitted separately, and the reported privacy budget is per-release without composition across communication rounds.

  3. Several important terms such as “operator space,” “high-frequency noise,” and “consensus update” would benefit from more precise definitions. The manuscript also contains unresolved references (“Fig. ??” / “Sec. ??”), duplicated text, and inconsistent method names such as “UMEDA” and “Diff-GNO”.

Questions for authors
  1. I'd like to know what neighboring-dataset definition the SP-DP analysis actually assumes, and why clipping to norm C implies sensitivity C rather than 2C. It would also help to clarify the privacy guarantee for the full client message, including Δθ_{\M}, and how the budget composes across communication rounds.

  2. What actually ensures that the semantic matrices coming from Wi-Fi, LiDAR, radar, and so on share the same latent coordinate system, rather than just happening to have the same d×d shape? And can the authors show evidence that modality-specific nuisance information really is concentrated in the spectral components that SGLT suppresses?

  3. Why should a reverse-diffusion sample from the learned update distribution be read as a consensus update, or as a useful descent direction for the federated objective? I'd also like the per-round score-network training procedure spelled out, and a comparison of Diff-NOA against simpler robust or deterministic aggregation rules applied in the same spectral coordinates.

Reviewer_uQs1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an operator-space federated learning framework for heterogeneous multimodal sensing. It integrates a Spectral-Gated Linear Transformer (SGLT) for shared semantic representation, Diffusion-Based Neural Operator Aggregation (Diff-NOA) for server-side update aggregation, and Subspace-Projected Differential Privacy (SP-DP) for privacy protection. Experiments on MM-Fi and RELI11D demonstrate promising results in pose estimation, action recognition, localization, and cross-dataset transfer.

Critical strengths

The paper addresses important challenges in heterogeneous multimodal federated learning, including inconsistent sensor representations, non-IID data, and privacy constraints. The proposed framework offers an interesting attempt to integrate these challenges within a unified formulation.

Critical weaknesses

1.The semantic matrix (M=\phi(K)^\top V) is computed from input-dependent representations, yet the federated algorithm treats it as a persistent model parameter with trainable updates. The paper does not sufficiently explain how these two interpretations are reconciled or how corresponding operators are aligned across heterogeneous clients. This ambiguity affects the foundation of the proposed operator-space aggregation.

2.SP-DP protects only the semantic operator updates, while the remaining parameter updates are transmitted and aggregated without an explicitly defined privacy mechanism. Although the paper acknowledges the per-release scope of its guarantee, it does not establish differential privacy for the complete federated training procedure. This substantially limits the practical privacy claims.

3.Diff-NOA samples a global update from a learned distribution of client updates, but the paper does not establish why such a sample should approximate an optimal federated aggregation direction. Although comparisons with FedAvg and partial-block averaging are provided, stronger alternatives such as robust or weighted spectral aggregation are needed to justify the additional complexity of diffusion modeling.

4.The main experimental protocol specifies a client participation rate of 0.4, while Table 3 reports 360 communication rounds to reach the localization target. However, Appendix Table 8 reports 160 rounds at the same participation rate and 360 rounds at a rate of 0.1. The differences in experimental configurations or reporting procedures should be explicitly clarified to ensure reproducibility.

Questions for authors

1.How is the input-dependent semantic matrix converted into a trainable federated parameter update, and how is semantic correspondence maintained across heterogeneous clients?

2.Why does the noise-allocation experiment include (\kappa=0.5) when the theoretical privacy guarantee requires (\kappa\geq1)? How does the subsequent spectral projection affect the injected noise?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.