MidGuard: Frequency-Aware Adversarial Protection for Personalized Diffusion Models

Diffusion models have achieved remarkable success in high-quality image synthesis, while personalization further enable them to reproduce personal identities from only a few reference images. Despite enabling customized applications, this capability raises privacy concerns through unauthorized identity replication and manipulation. Existing adversarial defenses optimize diffusion-based objectives to construct imperceptible perturbations, yet they face a trade-off between disrupting unauthorized identity learning and preserving image quality. Addressing this trade-off calls for update directions that effectively disrupt identity learning while introducing minimal visual distortion. To investigate how this balance varies across spatial scales, we decompose adversarial gradients into different frequency bands and compare the resulting updates under the same perturbation budget. In our analysis, low-frequency updates suppress identity learning but cause greater distortion, whereas high-frequency updates better preserve appearance but offer weaker protection. Mid-frequency updates achieve comparable suppression to low-frequency updates with less distortion. We therefore propose MidGuard, a frequency-aware adversarial protection method for personalized diffusion models. It fuses Gaussian band-pass-filtered mid-frequency gradients with raw gradients to retain complementary signals, while progressively decays the contribution of mid-frequency guidance within each inner optimization loop to enable less spectrally constrained refinement. Extensive experiments demonstrate stronger overall identity protection than representative adversarial defenses while maintaining competitive visual fidelity. Code is available at [https://anonymous.4open.science/r/luck](https://anonymous.4open.science/r/luck).