acceptodds
Under review as a conference paper at ICLR 2027

LoRA-based Data Augmentation for Federated Learning on Non-IID Data

Abstract

Federated learning (FL) enables privacy-preserving collaborative training. However, its performance degrades substantially under non-independent and identically distributed (non-IID) client data due to client drift or distribution skew. Existing methods often fail to provide distribution-aligned samples to local clients. We propose Federated Harmonization via Diffusion with Low-Rank Adaptation, Federated Harmonization via Diffusion with LoRA, or simply FedHDL, a generative framework that leverages a pretrained Stable Diffusion backbone with lightweight local LoRA adapters. Instead of aggregating full parameters, the server aggregates only LoRA adapters and performs cross-style synthesis to generate missing class–style samples. By using these synthetic samples for data augmentation, FedHDL effectively aligns local data with the global distribution, thereby re-balances label distributions and mitigates feature skew while maintaining low communication overhead. Extensive experiments across diverse non-IID scenarios show that FedHDL consistently improves upon prior generative augmentation methods while achieving competitive performance against strong federated optimization and personalization baselines. The experimental results demonstrate that parameter-efficient generative alignment is an effective path toward robust federated learning under distribution shift.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated Learning, Non-IID, Stable Diffusion, LoRA, Data Augmentation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2aSiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a novel method FedHDL for data augmentation in federated learning to address client drift and privacy concerns existing in federated learning. The method fine-tunes LoRA adapters attached to a frozen stable diffusion model and train them on client’s private data, which are uploaded to the server where the adapters are combined in a global adapter and used to generate samples. These generated samples combined with private client data are used to train the local client models which are aggregated using a SynCW-FedAvg method. Their method is verified with two forms of skewness in client data: label-skewness using the CIFAR-10 dataset, and feature-skewness using the Office-Home, PACS, and DomainNet datasets.

Critical strengths
  • The paper highlights the problem in non-IID distirbution among federated learning clients, which is a valid and strong problem to address.
  • The experimental design through label-skewness and feature-skewness is a good and fair way to evaluate the problem empirically.
Critical weaknesses
  • Privacy-preservation concerns:

    • The authors discuss the core problem from the perspective of privacy preservation and distribution skewness. However, their discussion avoids privacy-preservation methods in federated learning and overall deep learning. Particularly, privacy concerns in federated learning is a massive field, which the authors fail to address in their discussions.

    • The authors fail to provide any strong evidence for privacy preservation in their work. Particularly, federated learning methods, even without sharing raw data, are known to have privacy leakage in the literature [1,2,3]. While their federated aggregation differs from FedAvg with SynCW-FedAvg, it is still a modified version of FedAvg and does not address privacy leakage.

    • Diffusion models are also known to be prone to memorization issues, as literature have demonstrated near-exact training images sampled during generation time [4,5,6]. The proposed method directly uploads LoRA weights trained on the private client images to the server, and also uses the same stable diffusion model across clients. This raises the privacy concern even more as no preservation strategy is followed to to protect against adversarial servers.

  • Empirical evidences are not strong enough:

    • Their baseline experiments are limited for the label-skewed benchmark with only one dataset evaluated. To establish improvement in label-skewness, evaluations on more and diverse datasets are needed.

    • For the feature-skewness benchmarks, two out of three benchmark datasets do not show improvement with their method. Specifically, two out of the four domains in the PACS dataset underperform compared to existing benchmarks, while the average accuracy is also the second best compared to DualFed. On the other hand, for the DomainNet benchmark, their method outperforms only in a single domain, where the accuracy gain is only 0.10% over the second-best benchmark. It is unclear, given these concerns, how the method actually outperforms baselines to be established as major improvements.

  • Additional experiments outside of the core experiments are limited:

    • The authors report ablation studies for their core method. However, the main contribution regarding the mitigation of skewness is underexplored. Particularly, I think some form of experiments showing how the distribution benefits from their method is much needed. A simple and logical approach would have been a direct comparison of the data distributions before and after applying their methods, which could show the improvement of class distribution or feature distribution among clients.

    • To establish privacy preservation, as the authors have discussed in the earlier sections, attack methods such as membership inference, memorization attacks, and reconstruction attacks are needed to be demonstrated.

Overall, I believe this paper needs major revisions to be considered a solid contribution to the field.

References:

[1] Zhu, L., Liu, Z., & Han, S. (2019). Deep leakage from gradients. Advances in neural information processing systems, 32.

[2] Geiping, J., Bauermeister, H., Dröge, H., & Moeller, M. (2020). Inverting gradients-how easy is it to break privacy in federated learning?. Advances in neural information processing systems, 33, 16937-16947.

[3] Nasr, M., Shokri, R., & Houmansadr, A. (2019, May). Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP) (pp. 739-753). IEEE.

[4] Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramer, F., ... & Wallace, E. (2023). Extracting training data from diffusion models. In 32nd USENIX security symposium (USENIX Security 23) (pp. 5253-5270).

[5] Somepalli, G., Singla, V., Goldblum, M., Geiping, J., & Goldstein, T. (2023, June). Diffusion art or digital forgery? investigating data replication in diffusion models. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 6048-6058). IEEE.

[6] Somepalli, G., Singla, V., Goldblum, M., Geiping, J., & Goldstein, T. (2023). Understanding and mitigating copying in diffusion models. Advances in Neural Information Processing Systems, 36, 47783-47803.

Questions for authors

None.

Reviewer_TLnAOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The FedHDL method proposed by the authors aims to address the issue in federated learning where, if a client lacks data for certain categories or visual styles, simply modifying the aggregation algorithm cannot replenish the missing training information. FedHDL used federated LoRA to adapt Stable Diffusion for generating samples of categories and styles missing from clients to augment local data, while simultaneously optimizing the global classification model by using weighted FedAvg. FedHDL achieved significant performance improvements under severe non-IID conditions and validated the effectiveness of generative data augmentation in mitigating label and feature heterogeneity across multiple datasets.

Critical strengths

a. FedHDL demonstrated significant performance advantages under severe label skew. This showed generating training samples for missing classes could indeed yield substantial performance gains under conditions of severe class imbalance.

b. Data augmentation mechanism of FedHDL not only generated samples for missing classes but also utilized style-specific LoRA to generate images in diverse visual styles, simultaneously addressing label skew and feature skew with validation across different visual domains.

Critical weaknesses

a. The method had limited novelty and overlaped with existing research on generative Federated Learning and LoRA aggregation. In particular, the combination of federated LoRA and cross-style synthesis had not sufficiently demonstrated advantages over simply using Stable Diffusion for data augmentation. The core component cross-style synthesis lacked sufficient technical validation. While existing experiments demonstrate the effectiveness of generative augmentation, they are insufficient to independently validate the core mechanism of cross-style coordination.

b. The experimental section lacked relevant state-of-the-art generative baselines, and the rules for hyperparameter selection were unclear. The paper emphasized parameter efficiency and low communication overhead but did not provide a comprehensive resource comparison.

Questions for authors

a. Could the authors demonstrate that combining Federated LoRA with cross-style synthesis outperforms using Stable Diffusion directly for data augmentation through theoretical proof or experimental results?

b. The ablation study in Table 5 is limited to CIFAR-10 under label skew and does not independently validate the cross-style synthesis mechanism under feature heterogeneity. Could the authors provide additional ablation experiments on Office-Home, PACS, or DomainNet to demonstrate the effectiveness of this mechanism?

c. Could the authors explain how the server constructs the style-specific LoRA pool, and how the target style adapter was selected for different clients?

d. Regarding the parameter efficiency and low communication overhead emphasized by the authors, could they provide a comparison of total GPU training time, peak VRAM usage, total communication volume, or end-to-end execution time?

Reviewer_GUkLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes FedHDL, a diffusion-based augmentation method for non-IID FL. Each client trains a LoRA adapter on Stable Diffusion v1.5, the server averages the adapters in weight space and re-factorizes with a truncated SVD, and a client-specific mix of the global and a style adapter is used to generate class/style samples until every class matches the client's largest class. The classifier is then trained with FedAvg, where synthetic samples count with weight α in the aggregation. Experiments are on CIFAR-10 with Dirichlet label skew and on Office-Home, PACS and DomainNet with one domain per client.

Critical strengths
  1. Filling missing class/style combinations with a pretrained generator is a sensible way to go after non-IID data, and the paper is easy to follow.
  2. Aggregating and interpolating in the BA (weight update) space instead of on the LoRA factors is the right call.
  3. Four benchmarks covering label and feature skew, three seeds, and a reasonable set of optimization and personalization baselines.
Critical weaknesses
  1. Novelty is thin. SD-based augmentation of missing classes in non-IID FL is already Gen-FedSD (Morafah et al. 2024), weighted averaging of BA followed by truncated SVD is what FlexLoRA does (Bai et al., NeurIPS 2024), and interpolating LoRA updates is standard LoRA merging. Some close work isn't discussed, e.g. FedBiP (Chen et al., CVPR 2025), which personalizes a latent diffusion model for feature-skewed clients, and ReGL (arXiv 2409.14063), which uses generative models to recover the global distribution locally under label skew.

  2. Gen-FedSD is missing from Table 1, and that's where the headline gain is (71.4 vs 60.0). On CIFAR-10 there's no style at all (the prompt is just "a photo of a {label}"), so I suspect most of the gain comes from adding SD images of the missing classes, which Gen-FedSD already does. Without that baseline, and a plain SD baseline with no LoRA, Table 1 mainly shows that synthetic data helps.

  3. It looks like the main table picks the best LoRA rank per column. FedHDL's numbers in Table 1 (71.41, 71.57, 73.64, 73.36) match ranks 32, 64, 128 and 32 in Table 6. Table 5 also reports "best accuracy", and α = 0.6 was chosen "empirically". If these are selected on the test set, the comparison isn't fair to baselines that get one config. Also, Office-Home is 80.53 in Table 2 but the peak in Fig. 3 is 80.75.

  4. The feature-skew gains are small. FedHDL is +0.8 over DualFed on Office-Home, ties on PACS and loses by 1.8 on DomainNet, while using a 1B-parameter generator and a lot more training images. For Gen-FedSD the std is over the last three rounds, not seeds (Tables 3-4), so it isn't comparable to the other rows. With 4 clients and in-domain test sets, the cross-style claim isn't really tested; a leave-one-domain-out eval, or DualFed/FedSOL trained on the augmented data, would say more.

  5. Privacy and cost are claimed but never measured. The server keeps a style adapter per client, trained on that client's private images, and fine-tuned diffusion models are known to memorize training data (Carlini et al. 2023). That is pretty close to sharing a generator of the client's data, and there's no DP or attack analysis. It's also unclear who generates: the abstract and Fig. 1 say the server, Sec. 3.2 says the client. Either way, the bytes sent and the GPU time for tens of thousands of 28-step SD samples aren't reported, so "low communication overhead" is unsupported.

  6. Several details are missing or don't add up. λk is never given or ablated, and it's not said how the target style is picked or how many rounds the LoRA stage runs. The appendix says 10 of 100 CIFAR-10 clients per round but Sec. 4.3 says 5, and it's not explained how 4-domain PACS is split over 100 clients. The SVD step adds 1.7 points with std 2.29, on CIFAR-10 where there are no styles, so "filters out local domain noise" isn't supported. Minor: the method name is spelled out three times in the abstract, "By providing distribution-aligned samples, These results...", "11.4%" should be points, FedSAM is credited to both Qu and Caldarola, the FedBN PACS average (92.45) doesn't match its domains (90.95), and no generated images are shown anywhere.

Questions for authors
  1. Can you add Gen-FedSD and a plain SD (no LoRA) baseline to Table 1, with the same number of synthetic images?
  2. Are the Table 1 numbers from one fixed rank and final-round accuracy? What do you get with r = 64 everywhere?
  3. Who generates the images, how many per client, and what's the total communication and generation cost?
  4. Did you check whether the style adapters can reproduce client training images, e.g. with nearest-neighbor retrieval or a membership attack?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.