Privacy-Preserving Personalized Federated Latent Diffusion Models

Scaling differentially private federated diffusion models to high-dimensional images requires overcoming two barriers: increasingly damaging privacy noise and rising training costs. While latent diffusion models offer a promising way to address these barriers, the properties of their latent representations that favor private collaborative learning remain underexplored. We propose Personalized Federated Training of Latent Diffusion Models (PF-LDM): a shared server-side latent diffusion model learns cross-client structure from privatized latent representations, while client models refine generated samples to recover fine-grained local details. Our theory and experiments link privacy–utility gains to the distribution of latent representations, rather than to their reduced dimensionality alone. Specifically, we prove that PF-LDM satisfies local differential privacy for each client and derive utility guarantees for learning a Gaussian mixture model. These guarantees and empirical findings show how collaboration, simpler structures (e.g., fewer mixture components), and more discriminative latent features improve learning. Extensive experiments on CelebA-HQ, FFHQ, and the medical imaging dataset CheXpert show that PF-LDM generates high-fidelity images and improves performance on underrepresented classes across clients while protecting client data privacy.