Decomposing Private Image Generation via Coarse-to-Fine Wavelet Modeling

Generative models trained on sensitive image datasets have been shown to risk memorizing and reproducing individual training examples, making strong privacy guarantees essential. While differential privacy (DP) provides a principled basis for such guarantees, applying standard DP finetuning (e.g., with DP-SGD) to generative models often results in severe degradation of image quality. To address this, we introduce a spectral DP framework that uses wavelet representations to control which image scales are included in private finetuning, motivated by the hypothesis that dataset-specific information needed for adaptation, such as face geometry and object shape, can be captured by low-frequency wavelet components while fine-scale details can be supplied by a public prior. Specifically, we privately finetune a publicly pretrained autoregressive spectral image model to generate intermediaries at a chosen level of detail, then use the frozen public model to complete any remaining wavelet scales without additional privacy loss. Experiments with coarse-prefix and full-sequence configurations on MS-COCO and MM-CelebA-HQ demonstrate strong distributional image quality relative to diffusion and raster-autoregressive baselines. Full-sequence adaptation generally yields better final-resolution quality while coarse-prefix adaptation achieves stronger results at the coarse evaluation scale with lower training cost.