- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a federated learning framework for multimodal human sensing with heterogeneous sensors. It has three parts. (1) SGLT is a linear-attention encoder that applies a soft singular-value gate to the matrix M = φ(K)ᵀV. (2) Diff-NOA projects client updates of this "semantic block" onto the top-r singular directions of the global block, trains a small score model on the resulting r²-dimensional vectors each round, and uses a reverse-diffusion sample as the server update; all other parameters use FedAvg. (3) SP-DP clips the semantic update and adds Gaussian noise with a larger scale outside the top-r subspace. Experiments on MM-Fi report pose, action and localization results against many federated baselines, with a transfer test to RELI11D and several robustness sweeps.
Critical strengths
Relevant problem. Federated training across clients with different sensor sets, resolutions and missing modalities, under privacy constraints, is a real need in device-free sensing, and the paper tries to evaluate all of these stressors in one setting.
Scope of the privacy claim is stated. The paper says the guarantee is per release and covers only the semantic update, which at least makes the limits of the claim visible.
Critical weaknesses
[The method is not well defined, and its three components are not justified]
-
What is aggregated?: M = φ(K)ᵀV is an activation that changes with every input, yet the paper treats θ_M as a parameter block with client updates ΔM_k and a server-side SVD. How an input-dependent matrix becomes a federated parameter is never explained. The stated motivation, that clients with different token counts cannot be aggregated, also does not hold: transformer weights do not depend on sequence length, so FedAvg already applies. The "discretization invariance" lemma only states that a d×d matrix is d×d.
-
Why diffusion?: Aggregation needs a point estimate, but Diff-NOA draws one sample from a model of the client-update distribution. The score model is trained on about 40 vectors of dimension 256 per round, a regime where it can do little more than memorize them, so the sample is close to one client's update. No argument is given for why a sample should beat the mean, a median or a trimmed mean. The interaction with DP makes this harder to accept: by my rough calculation from Eq. (9), at ε = 2 and C = 1 the noise has σ ≈ 2.4 per coordinate, so after projection the noise norm is on the order of 40 while the clipped signal norm is at most 1. Averaging over clients is what reduces such noise; sampling does not. Yet the paper reports that sampling outperforms averaging in the same coordinates (Table 9) and that privacy costs only 1.9 mm in MPJPE (Table 1).
-
Neural operator: Nothing in the method learns a map between function spaces. The term refers to a 2-layer MLP score network over a 256-dimensional vector.
-
SP-DP: Theorem B.4 shows the mechanism is isotropic Gaussian noise at scale σ_sig plus extra noise in the complementary subspace. It is therefore never less noisy than the standard Gaussian mechanism at the same ε, and any utility gain must come from the server's projection, which is post-processing available to the isotropic baseline too. As I read Eq. (10), that projection removes the extra noise exactly, so κ should have no effect on the aggregated update, yet Table 7 reports large effects of κ. The text is also inconsistent about where noise goes ("confine noise to the signal's null space" vs. noise in both subspaces). The calibration in Eq. (9) is the classical bound, which is stated for ε < 1 (see Balle & Wang, ICML 2018) but is used up to ε = 8.
-
Privacy scope: Only ΔM is privatized; all other parameter updates are uploaded in the clear, and the guarantee is per round over hundreds of rounds with no composition. Comparing "at matched ε" with DP-FedAvg, which perturbs the whole model, is not a like-for-like comparison, and the gradient-inversion test attacks only the protected block.
[The empirical evidence is internally inconsistent and the positioning is weak]
-
Inconsistent numbers: In Table 1 the "LAT (no gate)" baseline and the "w/o SGLT (LAT)" ablation have identical means on all three tasks (89.8 / 87.4 / 12.2) with different standard deviations, although one is a plain federated baseline and the other should include Diff-NOA and SP-DP. Table 3 reports 360 rounds to target at q = 0.4, while Table 8 in the appendix reports 160 rounds at q = 0.4 and 360 at q = 0.1. The main text uses T = 1000 rounds and the appendix T = 200.
-
Missing or undefined baselines: CA2FL, HPFL and Decoupled Subgraph FL are listed as baselines but appear in no table. FedRod, FedGCN and FedGAT appear in tables without description. It is unclear how graph-based FL methods apply to this task.
-
Protocol gaps: How 100 clients are formed from MM-Fi, which modalities each client type holds, how the modality-specific encoders are shared, and how localization is evaluated on RELI11D without adaptation are not specified.
-
Related work: Diffusion-based server aggregation exists (pFedGPA, Lai et al., AAAI 2025). Projecting private gradients onto a public low-dimensional subspace before or after adding noise is established (Zhou et al., ICLR 2021; GEP, Yu et al., ICLR 2021). Multimodal FL with heterogeneous or missing modalities has dedicated methods and benchmarks (Harmony, Ouyang et al., MobiSys 2023; CreamFL, Yu et al., ICLR 2023; FedMultimodal, Feng et al., KDD 2023). None are cited or compared, while the references given for "generative FL" include a paper on DP convergence. The comparison is instead filled with generic FL optimizers.
Questions for authors
What exactly is θ_M? Which trainable parameters does it contain, and how is a client "update" ΔM_k of an input-dependent matrix computed?
Why is a single diffusion sample a better server update than the mean, coordinate-wise median or trimmed mean of the same projected vectors? Please report these baselines under identical DP noise, and the variance of the server update across rounds.
Is the score model re-initialized each round or carried over? With about 40 training vectors, how do its samples differ from the training points?
What is the signal-to-noise ratio of the projected update at ε = 2? How can the utility loss from privacy be this small?
Given the projection in Eq. (10), through which path does κ change the result in Table 7?
Are the non-semantic parameter updates privatized in any experiment? If not, what does "matched ε" mean relative to DP-FedAvg?
Please explain the identical rows in Table 1 and the discrepancy between Table 3 and Table 8, and report the listed baselines that are missing from the tables.
How are clients, modalities and encoders assigned, and how is RELI11D localization evaluated?
Minor: several references to papers from 2020 onward give the venue as "NIPS" ([24], [32], [42], [46]); these should read NeurIPS. The manuscript header ("ICLR 2026"), the unresolved references ("Fig. ??", "Sec. ??") and the duplicated paragraphs in Sections 4.1 and 4.2 should also be fixed.