Invisible Theft: Memorization-Based Client-to-Client Data Leaks in Federated Diffusion Models

Federated diffusion has emerged as a promising framework for collaboratively training visual generation models without sharing private training data, making it particularly valuable for data-scarce yet privacy-sensitive federated settings. However, we reveal a previously overlooked client-to-client privacy vulnerability in this framework: because diffusion models can memorize and reproduce their training samples, a malicious client may exploit the shared global model to generate images highly similar to other clients' private training data, without accessing their gradients or interfering with the federated training process. To demonstrate the severity and practicality of this vulnerability, we propose a memorization-residual guided data stealing framework. Our key intuition is to leverage two models naturally accessible to the attacker—an attacker-private model trained only on its own data and the public pretrained model—as negative references, and use their prediction residuals with the global model as sampling-time guidance signals. This residual-guided sampling substantially increases the likelihood that the global model generates images highly similar to private training samples from other clients. Experiments show that our attack consistently increases both the frequency of high-similarity reproductions and the coverage of distinct threat images. Moreover, our attack is performed entirely after federated training and requires no modification to the training protocol, making it naturally stealthy from the perspective of federated training.