- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates federated learning (FL) under feature skew and superposition. Specifically, it first provides theoretical analyses highlighting the limitations of FedAvg in preserving feature representations and preventing cross-client interference under these settings. Based on these observations, the paper proposes MechLeash-FL, which regularizes model update directions using a sparse autoencoder (SAE) to capture feature representations. By alternating between updating the task models and the SAE, MechLeash-FL achieves improved reconstruction performance on both synthetic and real-world datasets.
Critical strengths
- The paper provides theoretical analyses that motivate the design of MechLeash-FL, making the proposed approach more theoretically grounded.
Critical weaknesses
-
The evaluation is limited to synthetic datasets and the relatively small-scale Office-Caltech10 dataset, focusing exclusively on reconstruction tasks. It remains unclear whether the reported improvements can generalize to larger-scale and more realistic datasets, as well as other downstream tasks, such as classification, object detection, and semantic segmentation, which are commonly studied in federated learning.
-
The idea of leveraging latent feature representations to regularize local training in federated learning has been explored in previous studies [1][2]. However, the paper provides insufficient comparisons and discussion with these existing approaches, making it difficult to assess the novelty and advantages of the proposed method.
[1] 2024 ICLR FedImpro: Measuring and Improving Client Update in Federated Learning
[2] 2024 NeurIPS Classifier Clustering and Feature Alignment for Federated Learning under Distributed Concept Drift
Questions for authors
Besides the weakness shown in the above section, please also see the following questions:
Q1: Since the SAE is trained to capture the distributions of local feature representations and is shared across clients, could this potentially introduce privacy risks or lead to information leakage, thereby compromising data privacy in federated learning?
Q2: The two-phase training procedure in MechLeash-FL may introduce additional computational overhead and increase training time. How significant are these additional costs relative to the performance improvements achieved? It would be valuable to report and discuss the trade-off between computational efficiency and performance gains.
3 comments
Details
Details for the answers above, under the same numbers. p-values are exact one-sided paired Wilcoxon tests over 10 seeds. "x/10" counts the seeds in which MechLeash-FL is better.
1. Larger datasets
Second dataset: On DomainNet (10-class subset, six domains as six clients, the paper's model and coefficients, 8 rounds), MechLeash-FL lowers the MSE by 22.9 % (FedAvg), 17.5 % (FedProx), 7.5 % (MOON) and 20.4 % (Ditto, personalized), all significant after Holm correction (all p < 0.05; MOON 8/10 seeds, the others 10/10).
More clients: we split each Office-Caltech10 domain into random parts. FedAvg + MechLeash-FL versus FedAvg:
| Clients | 4 | 8 | 16 | 32 |
|---|---|---|---|---|
| lambda_rec | 0.3 | 0.1 | 0.03 | 0.1 |
| MSE change | -30.4 % | -27.6 % | -10.5 % | -12.5 % |
| Seeds better, p | 10/10, 0.001 | 10/10, 0.001 | 9/10, 0.042 | 10/10, 0.001 |
The 4-client column repeats the paper's setting on new seeds. For 8 to 32 clients, lambda_rec was chosen on three separate seeds; the table shows 10 new seeds. All gains hold after Holm correction.
2. Classification
Office-Home (65 classes, 15.5k images, four domains as clients) with the paper's encoder, a 64-dimensional bottleneck, and a linear layer on top. Validation accuracy in %:
| FedAvg | FedProx | MOON | Ditto (personalized) | |
|---|---|---|---|---|
| lambda_rec | 0.3 | 1 | 1 | 0.3 |
| Plain | 19.0 | 18.0 | 16.6 | 25.4 |
| + MechLeash-FL | 20.9 | 23.4 | 24.7 | 25.5 |
| Seeds better, p | 7/10, 0.019 | 10/10, 0.001 | 10/10, 0.001 | 4/10, 0.46 |
The gains for FedAvg, FedProx and MOON are significant after Holm correction. In Ditto, the leash acts only on the global model, and the personalized accuracy does not change significantly. Accuracies are around 20 % because the paper's small encoder is trained from scratch on 65 classes (chance is 1.5 %). With a ResNet-18 on DomainNet-126, the method has no effect (accuracy within 0.3 points of FedAvg). We think this network has enough dimensions to store its features without superposition. So the method helps when the model has to store more features than it has dimensions, and we will add this to the limitations.
3. Detection and segmentation
Segmentation: prostate MRI, six hospitals as clients, the paper's encoder-decoder with per-pixel class outputs. The Dice score (overlap of predicted and true prostate area, 100 % is perfect) rises from 62.9 % (FedAvg) to 67.2 % (10/10, p = 0.001), and at every hospital.
Detection: the paper's encoder-decoder with a detection head was too weak (FedAvg at most 12 % mAP, mean average precision). A pretrained ResNet-18 with a 256-dimensional layer for the SAE reaches up to 84 % mAP, but the SAE uses fewer than 100 of its 512 feature directions, so we see no sign of superposition. MechLeash-FL showed no benefit with either.
4. FedImpro and FedCCFA
FedImpro trains the upper part of the model on features sampled from a class-conditional feature distribution estimated across clients. FedCCFA groups the per-class classifiers of the clients and aligns client features to class-level anchors to handle differences in P(Y|X) between clients. Both work with labels and class-level statistics. The leash of MechLeash-FL uses no labels and acts on the directions of individual features. We will discuss this in the revised paper.
5. Privacy
Each client trains the SAE locally, and the server only averages its parameters, as for the task model. No activations or images leave a client. The SAE parameters depend on local data as the encoder weights do, so the risk is of the same kind as for the model parameters FedAvg already shares. Protections for model updates, such as secure aggregation and differential privacy on the client side, can be applied to the SAE update in the same way.
6. Cost
Training time per run on the paper's Office-Caltech10 runs (one A100 GPU) and MSE reduction:
| FedAvg | FedProx | MOON | Ditto | |
|---|---|---|---|---|
| Time without → with, min | 15.1 → 18.6 | 18.8 → 22.4 | 15.9 → 20.7 | 34.0 → 38.3 |
| Extra time | +23 % | +19 % | +31 % | +13 % |
| Validation MSE reduction | 29 % | 33 % | 15 % | 31 % |
Communication grows by a factor of 1.55 per round. With the same total upload (30 rounds with MechLeash-FL, 48 without, Figure 6), the validation MSE is still 12.7 to 33.6 % lower. Two ways to reduce the cost:
| SAE budget (FedAvg) | Upload per round vs. FedAvg | Validation MSE reduction | Seeds better, p |
|---|---|---|---|
| 20 SAE epochs per round (paper setting) | 1.55x | 30.4 % | 10/10, 0.001 |
| 10 / 5 / 2 / 1 epochs | 1.55x | 29.8 / 29.4 / 28.2 / 23.0 % | 10/10, 0.001 each |
| Refit every 4 / 12 rounds | 1.14x / 1.05x | 23.0 / 15.2 % | 10/10, 0.001 / 7/10, 0.014 |
| Fitted once (SAE-Frozen) | about 1x | 10.4 % | 7/10, 0.042 |
Refitting only every r rounds skips the fit in r-1 of r rounds, and clients keep the last averaged SAE in between. The first two rows use new seeds (paper: 29.1 %), the last two use the paper's seeds.
Dear Reviewer,
Thank you again for your review. We uploaded the revised paper on 13 Nov (AoE). It includes the changes described above. If any questions or concerns remain, please let us know and we will gladly address them.
Best regards, The authors
New experiments, FedImpro/FedCCFA, privacy and cost
Thank you for the careful review and the two references. Below, we answer each point briefly. A second message, posted as a reply to this one, gives the details, tables and statistics under the same numbers. We will upload the revised paper by the 13th (AoE). The (➔...) marks show where each change goes.
1. Larger and more realistic datasets
We agree with this point and therefore ran DomainNet reconstruction, Office-Home classification (15.5k images, point 2) and runs with up to 32 clients on Office-Caltech10. The gains hold on those datasets, except for Ditto's personalized accuracy on Office-Home. On Office-Caltech10, they become smaller with more clients but stay significant. (➔Appendix)
2. Classification
We ran classification on Office-Home (65 classes, four domains as clients). MechLeash-FL raises the accuracy of FedAvg, FedProx and MOON by 1.9, 5.5 and 8.1 percentage points and leaves Ditto's personalized accuracy unchanged. (➔Appendix)
3. Detection and segmentation
We ran both. On prostate MRI segmentation with six hospitals as clients, MechLeash-FL raises the Dice score of FedAvg from 62.9 to 67.2 %. On object detection, we saw no benefit with the architectures we tried: the models were either too weak for the task or showed no sign of superposition, and in tasks without a sign of superposition, MechLeash-FL shows no benefit. We will add this to the limitations. (➔Appendix, Section 8)
4. FedImpro and FedCCFA
Thank you for these two references, which we had missed. We will discuss both in the related work. Like MechLeash-FL, both use latent features to guide local training. They do so with class-level statistics of the features, and MechLeash-FL with the directions of individual features. (➔Section 2)
5. Privacy
This is an important question we will discuss in the revision of the paper. The SAE adds a risk of the same kind as the model parameters FedAvg already shares: only its parameters are shared, and the same protections apply. (➔Section 5.3)
6. Cost versus gain
We will report it in the revised paper: depending on the baseline, 13 to 31 % more training time and 15 to 33 % lower validation MSE. Two cheaper variants keep most of the gain. (➔Appendix)
If you have further questions, we are happy to answer them.