acceptodds
Withdrawn from ICLR 2027

Privacy-Preserving Personalized Federated Latent Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Scaling differentially private federated diffusion models to high-dimensional images requires overcoming two barriers: increasingly damaging privacy noise and rising training costs. While latent diffusion models offer a promising way to address these barriers, the properties of their latent representations that favor private collaborative learning remain underexplored. We propose Personalized Federated Training of Latent Diffusion Models (PF-LDM): a shared server-side latent diffusion model learns cross-client structure from privatized latent representations, while client models refine generated samples to recover fine-grained local details. Our theory and experiments link privacy–utility gains to the distribution of latent representations, rather than to their reduced dimensionality alone. Specifically, we prove that PF-LDM satisfies local differential privacy for each client and derive utility guarantees for learning a Gaussian mixture model. These guarantees and empirical findings show how collaboration, simpler structures (e.g., fewer mixture components), and more discriminative latent features improve learning. Extensive experiments on CelebA-HQ, FFHQ, and the medical imaging dataset CheXpert show that PF-LDM generates high-fidelity images and improves performance on underrepresented classes across clients while protecting client data privacy.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Diffusion Models, Differential Privacy, Federated Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_LYuCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PF-LDM, a framework for training latent diffusion models with local differential privacy, targeting the loss of image quality that privacy noise causes in high-dimensional pixel space. In PF-LDM, each client encodes its images with a shared, fixed public autoencoder, clips the latents, adds Gaussian noise once at a chosen diffusion timestep, and sends the privatized latents to a server. The server trains a global denoiser on the pooled noisy latents, while each client keeps a local denoiser trained on its own clean latents. At generation time, the global model produces a sample at the noise level used for privatization, and the local model refines it to recover client-specific details. The paper provides local differential privacy guarantee and utility analysis for learning Gaussian mixture models. Experiments on face and chest X-ray datasets show that PF-LDM has better generation quality for underrepresented groups, better downstream classification, and near-chance membership inference against the shared model.

Critical strengths
  • Private training of generative models for sensitive domains such as medical imaging is an important problem. Releasing privatized latents once, with a guarantee that does not depend on the training length and without a trusted server, is a practical setting.
  • The paper is generally readable, and the pipeline of private release, global training and local refinement is easy to follow.
Critical weaknesses
  • The idea of privatizing data once with the forward diffusion noise and using the noisy samples for server-side training, together with the collaborative sampling that hands off to a local model, comes from PFDM (Patel et al., 2026). The main change here is applying it in the latent space of a pretrained autoencoder, which is a fairly incremental step.
  • The theory and the empirical explanation of "what favors collaboration" are loosely connected to the method. The utility result assumes that all clients draw from the same Gaussian mixture, which does not match the heterogeneous setting that motivates personalization, and the benefit of collaboration is stated under an assumption that the global model is more accurate at the handoff level.
  • The experimental evidence for the utility claims is not sufficient. There are only two clients with a hand-crafted imbalance, and the only baselines are the non-private and non-collaborative extremes. The most relevant private PFDM and DP-SGD baselines are missing. The FID gains are also mixed, since the non-collaborative baseline is better on the majority group or on average in several settings.
  • The practical benefit is not yet convincing. A single, fairly large privacy level is used, so the privacy–utility trade-off is not actually shown, so the guarantee is meaningful only at a fairly large privacy budget. The gains over a non-collaborative model are modest in many cases. Without a comparison to PFDM and DP-SGD baselines, it is hard to judge how much is gained by moving to latent space.
Questions for authors
  • How does PF-LDM compare to the PFDM method it extends and to a DP-SGD-based approach at the same privacy level? Without these baselines, it is hard to attribute the gains to latent-space privatization.
  • How does the performance change with the privacy budget (and the corresponding clipping radius and noise timestep)? The results use a single, fairly large budget, so it is unclear how the method behaves at stronger privacy.
  • The utility theorem assumes that all clients share the same latent distribution, while the experiments and motivation are about heterogeneous clients. Can the analysis be extended to heterogeneous distributions?
Reviewer_dTSeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a personalized federated latent diffusion framework, PF-LDM, where clients upload privacy-preserving latent representations while ahving local denoisers for refinement. The authors provide a per-sample local differential privacy guarantee and analyze utility under a shared Gaussian-mixture latent model. Empirical evaluations include image quality comparison, subgroup performance, reconstruction, candidate identification, and membership inference on face and medical datasets.

Critical strengths

The authors provide an explicit per-sample privacy mechanism and a clear post-processing statement with the shared denoiser. The theory usefully connects collaborative gains to the accuracy of the global model at the handoff point. The addtional analyses on privacy perspective is useful for verifying the algorithm's practical utilty.

Critical weaknesses

Novelty

  • The main concern is the novelty, relative to Patel et al. (2026).
    • It reads that the overall structure of the proposed method is largely inherited from PF-DM.
    • For example, PF-LDM retains the PF-DM's private local denoisers, one-time clipped and noised client release, pooled server-side global denoiser, and global-to-local sampling procedure.
    • Thus, I perceived as the main algorithmic change is to execute PF-DM in the latent space of a fixed public denoiser rather than directly in pixel space as PF-DM did.
      • While I acknowledge that it still does add meaningful latent-space contributions, including representation comparisons and broader evaluation with privacy/attack analysis, these contributions extend an existing collaborative architecture rather than a new one.
    • It seems necessary to sharply separate between inherited protocol and new technical contributions.
      • For instance, the authors should have provided a step-by-step PF-DM vs. PF-LDM correspondence, along with PF-DM baseline comparisons.
      • I think, without this, improvements caused by the latent representation cannot be confirmed from changes in datasets, models, and training configurations.

Privacy statement

  • The stated privacy guarantee does not yet describe the complete training protocol.
    • It is unclear whether repeated uploads, correlated releases, or multiple training rounds require composition beyond the per-sample guarantee.
    • It is also unclear whether any shared artifact depends on unclipped latents, gradients, optimizer states, metadata, or refinement outputs.

Theory-experiments gap

  • The theoretical setting seems considerably narrower than the experimental setting.
    • For example, it assumes shared equal-weight Gaussian mixtures with identity covariance.
      • How about exploring a bridge that Gaussian mixtures can approximate any distributions at some degree to possibly extend the current utility statement to arbitrary target distribution? (note: my two cents)
    • Its collaboration gain also depends on the global model already being more accurate at the handoff point.

Attack protocol

  • I think the empirical privacy evaluation didn't cover strong enough attackers.
    • While the authors acknowledged in Appendix E.1, some baselines, e.g., shadow models, uploads, and attacks against client refinement models should have been touched as they are close to practical attack scenario.
    • The reconstruction experiments themselves do not establish security against the strongest feasible attacker.
Questions for authors
  • Please consider adding in-depth technical and empirical differences between PF-LDM and Patel et al. (2026), in order to isolate novelty and motivation.
  • Please also cite and add comparisons with missing but relevant precursors that handled (private) FL for diffusion models:
    • Training Diffusion Models with Federated Learning (arXiv 2024)
    • VQ-FedDiff: Federated Learning Algorithm of Diffusion Models With Client-Specific Vector-Quantized Conditioning (IEEE TPAMI 2025)
    • Diffusion Federated Dataset (NeurIPS 2025)
  • Please explain how privacy composes across FL rounds, repeated uploads, and correlated releases from the same sample.
  • Doesn't any shared artifact depend on unclipped latents, gradients, metadata, or refined outputs?
  • How should the handoff level be set in practice, and how often does the required handoff accuracy condition hold across clients under and privacy budgets?
  • Please consider increasing font size in all figures.
Reviewer_A94qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes PF-LDM: a framework in which clients encode their images using a fixed public autoencoder and upload clipped, Gaussian-noised latent representations once. A server trains a global diffusion model on these privatized latents. During generation, a client uses the global model to produce a noisy latent and then refines it with its locally trained denoiser. The paper provides a sample-level local DP guarantee for the shared model and a utility analysis under Gaussian mixture assumptions. The framework is experimentally evalauted on CelebA-HQ, FFHQ, and CheXpert.

Critical strengths
  1. PF-LDM releases clipped, Gaussian-noised latent representations once and trains the shared model entirely through post-processing, avoiding additional privacy expenditure during server training. The sample-level local DP guarantee protects image contents before they reach the server, while local denoisers support personalized refinement. This is important as it provides a useful approach to collaborative generation without requiring a trusted server.

  2. PF-LDM improves minority-group FID over non-collaborative training across all nine dataset–architecture settings in Table 1 on CelebA-HQ, FFHQ, and CheXpert datasets. Also, it improves all reported downstream accuracy and F1 comparisons in Table 2. These findings demonstrate the practical value of collaboration when individual clients have very limited data for particular groups.

Critical weaknesses
  1. Table 1 provides no variability estimates across independent diffusion-model training runs. Although several improvements over non-collaborative training are promising, others are small: average FID changes from 22.7296 to 22.5686 for SoftVQ-VAE on CelebA-HQ and slightly worsens on CheXpert. Independent runs varying training and privatization randomness, with uncertainty estimates, are needed to determine which gains are reproducible and substantiate the benefits.

  2. Table 1 compares PF-LDM against non-collaborative training and non-private centralized training, but includes no competing private collaborative method. Specifically, a matched comparison with pixel-space PFDM,the framework that PF-LDM directly extends, seem to be missing. Thus, although the experiments demonstrate benefits from collaboration over local-only training, they do not establish the empirical advantage of latent-space privatization over PFDM or position PF-LDM relative to existing private generation methods.

Minor comment: I could not find a code availability statement or repository link in the manuscript. If not already provided in the supplementary material, including anonymized code and configuration files would facilitate reproducibility and verification of the experimental results.

Questions for authors
  1. I was wondering if you have evaluated PF-LDM against pixel-space PFDM under matched privacy guarantees, data splits, and comparable computational budgets? If so, please provide the results. If not, what prevents this comparison, and what evidence supports an empirical advantage over PFDM?

  2. Do the Table 1 results represent single diffusion-model training runs or averages across independent runs? If repeated runs are available, please report variability and uncertainty in the PF-LDM–NC differences, accounting for both training and privatization randomness. This is particularly relevant to the small improvement on CelebA-HQ and slight deterioration on CheXpert with SoftVQ-VAE.

  3. Footnote 1 states that clipping is omitted for SoftVQ-VAE because the evaluated latent norms are below the chosen radius. Is clipping nevertheless retained as a safeguard for arbitrary inputs, or does the encoder provide a provable bound on latent norms? Please clarify how the sensitivity bound required by Theorem 4.1 is ensured.

  4. Table 10 uses different sampling procedures and denoising-step budgets for PF-LDM and NC. How sensitive are the reported improvements to these choices, and are comparisons at matched denoising-evaluation budgets available?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.