High Performance Differentially Private Fine-Tuning using Dataset Distillation

Differentially Private Stochastic Gradient Descent (DP-SGD) is the dominant approach to private deep learning, but it pays compute and privacy budget per trained model: each new architecture, ensemble member, or downstream fine-tuning round incurs additional privacy cost. We propose SPS (Summarize–Privatize–Synthesize) and its enhanced variant SPS+, dataset-distillation algorithms that release a private synthetic dataset by privatizing intermediate activation statistics from a public pretrained model. Once released, the dataset is post-processed freely: any downstream architecture, ensemble, federated aggregation, or continual update incurs zero additional privacy cost. In particular, a single SPS+ dataset distilled from a Wide ResNet-22-8 transfers zero-shot to architectures with different inductive biases (Vision Transformers, Swin Transformers, and ConvNeXt), all without re-incurring privacy. Empirically, SPS+ is competitive with state-of-the-art DP-SGD across $ϵ\in \{1,2,4,8\}$ on CIFAR-10 and CIFAR-100, with a notable advantage on CIFAR-100 under strict privacy, achieving a $5.8\%$ gain at $ϵ=1$ over compute-matched DP-SGD. SPS+ additionally outperforms prior generation-based DP methods by a large margin and supports private federated and continual learning out of the box.