- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
In this paper, the authors identify a client-to-client privacy vulnerability in federated diffusion. Since the global model is technically fine-tuned on every client's data and diffusion models are known to memorize samples, a malicious client can regenerate other clients' private training images from the distributed global model. The malicious clients are able to do so without observing their gradients, accessing the server, or interfering with training. The challenge faced by this malicious attacker is entangled memorization, where the global model's memorization gets mixed with the clients' data, the attacker's own data, and the pretraining prior. The last two are unwanted. The attacker has access to its own data and the public pretrained checkpoint, so it is able to construct two negative guidance residuals at sampling time to get rid of unwanted information. Overall, I really like the idea, presentation, and implementation of the paper. I have some minor concerns and questions, which I have listed below.
Critical strengths
-
The threat model is novel and well motivated. The consequence is alarming in a useful way; in a medical federated setting, every participating hospital could become a serious attacker.
-
The decomposition of the global model's memorization is done very elegantly into three sources. Two of these sources contribute to the negative-guidance terms, thereby isolating the contribution of other clients' data from the attacker's own data and the pretraining knowledge of the global model.
-
Overall, the paper is a good read. It is well structured and easy to understand. The experiments and ablations are solid, and I don't really have any major concerns.
Critical weaknesses
-
It is not clear which parameters are fine-tuned. The paper mentions that the model is initialized from Stable Diffusion and fine-tuned. Unless I have completely missed it, it is not mentioned anywhere in the paper whether it is full fine-tuning, U-Net only, U-Net plus text encoder, or some other LoRA scheme.
-
The client count and the round budget should be in the main text. The number of clients is never given as a proper number, and the reader has to infer K=4 for Fed-Fundus and K=6 for Fed-Prostrate. The round budgets appear in the Appendix too and should be in the main text since that is something a reader wants to know when reading about an FL experiment.
-
Figure 3 was hard to understand for me. a) and b) regions are not visually separated, so it is unclear where one ends and the other begins. The color scheme is a bit confusing as well. The b) label is red, and various elements in the figure are red as well. Is there a correlation between b) and the red part of the figure?
Questions for authors
-
Is the federated fine-tuning for the full model, or is it some LoRA configuration? If it is full fine-tuning, does the attack retain its effectiveness under various LoRA ranks?
-
Why are there 5 rounds for Fed-Fundus vs. 25 rounds for Fed-Prostate?
-
Does the attack still work if the clients adopt a memorization-mitigation strategy or DP noise?
-
How does the attack success scale with increasing the number of clients?