- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces a novel method FedHDL for data augmentation in federated learning to address client drift and privacy concerns existing in federated learning. The method fine-tunes LoRA adapters attached to a frozen stable diffusion model and train them on client’s private data, which are uploaded to the server where the adapters are combined in a global adapter and used to generate samples. These generated samples combined with private client data are used to train the local client models which are aggregated using a SynCW-FedAvg method. Their method is verified with two forms of skewness in client data: label-skewness using the CIFAR-10 dataset, and feature-skewness using the Office-Home, PACS, and DomainNet datasets.
Critical strengths
- The paper highlights the problem in non-IID distirbution among federated learning clients, which is a valid and strong problem to address.
- The experimental design through label-skewness and feature-skewness is a good and fair way to evaluate the problem empirically.
Critical weaknesses
-
Privacy-preservation concerns:
-
The authors discuss the core problem from the perspective of privacy preservation and distribution skewness. However, their discussion avoids privacy-preservation methods in federated learning and overall deep learning. Particularly, privacy concerns in federated learning is a massive field, which the authors fail to address in their discussions.
-
The authors fail to provide any strong evidence for privacy preservation in their work. Particularly, federated learning methods, even without sharing raw data, are known to have privacy leakage in the literature [1,2,3]. While their federated aggregation differs from FedAvg with SynCW-FedAvg, it is still a modified version of FedAvg and does not address privacy leakage.
-
Diffusion models are also known to be prone to memorization issues, as literature have demonstrated near-exact training images sampled during generation time [4,5,6]. The proposed method directly uploads LoRA weights trained on the private client images to the server, and also uses the same stable diffusion model across clients. This raises the privacy concern even more as no preservation strategy is followed to to protect against adversarial servers.
-
-
Empirical evidences are not strong enough:
-
Their baseline experiments are limited for the label-skewed benchmark with only one dataset evaluated. To establish improvement in label-skewness, evaluations on more and diverse datasets are needed.
-
For the feature-skewness benchmarks, two out of three benchmark datasets do not show improvement with their method. Specifically, two out of the four domains in the PACS dataset underperform compared to existing benchmarks, while the average accuracy is also the second best compared to DualFed. On the other hand, for the DomainNet benchmark, their method outperforms only in a single domain, where the accuracy gain is only 0.10% over the second-best benchmark. It is unclear, given these concerns, how the method actually outperforms baselines to be established as major improvements.
-
-
Additional experiments outside of the core experiments are limited:
-
The authors report ablation studies for their core method. However, the main contribution regarding the mitigation of skewness is underexplored. Particularly, I think some form of experiments showing how the distribution benefits from their method is much needed. A simple and logical approach would have been a direct comparison of the data distributions before and after applying their methods, which could show the improvement of class distribution or feature distribution among clients.
-
To establish privacy preservation, as the authors have discussed in the earlier sections, attack methods such as membership inference, memorization attacks, and reconstruction attacks are needed to be demonstrated.
-
Overall, I believe this paper needs major revisions to be considered a solid contribution to the field.
References:
[1] Zhu, L., Liu, Z., & Han, S. (2019). Deep leakage from gradients. Advances in neural information processing systems, 32.
[2] Geiping, J., Bauermeister, H., Dröge, H., & Moeller, M. (2020). Inverting gradients-how easy is it to break privacy in federated learning?. Advances in neural information processing systems, 33, 16937-16947.
[3] Nasr, M., Shokri, R., & Houmansadr, A. (2019, May). Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP) (pp. 739-753). IEEE.
[4] Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramer, F., ... & Wallace, E. (2023). Extracting training data from diffusion models. In 32nd USENIX security symposium (USENIX Security 23) (pp. 5253-5270).
[5] Somepalli, G., Singla, V., Goldblum, M., Geiping, J., & Goldstein, T. (2023, June). Diffusion art or digital forgery? investigating data replication in diffusion models. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 6048-6058). IEEE.
[6] Somepalli, G., Singla, V., Goldblum, M., Geiping, J., & Goldstein, T. (2023). Understanding and mitigating copying in diffusion models. Advances in Neural Information Processing Systems, 36, 47783-47803.
Questions for authors
None.