acceptodds
Withdrawn from ICLR 2027

MidGuard: Frequency-Aware Adversarial Protection for Personalized Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion models have achieved remarkable success in high-quality image synthesis, while personalization further enable them to reproduce personal identities from only a few reference images. Despite enabling customized applications, this capability raises privacy concerns through unauthorized identity replication and manipulation. Existing adversarial defenses optimize diffusion-based objectives to construct imperceptible perturbations, yet they face a trade-off between disrupting unauthorized identity learning and preserving image quality. Addressing this trade-off calls for update directions that effectively disrupt identity learning while introducing minimal visual distortion. To investigate how this balance varies across spatial scales, we decompose adversarial gradients into different frequency bands and compare the resulting updates under the same perturbation budget. In our analysis, low-frequency updates suppress identity learning but cause greater distortion, whereas high-frequency updates better preserve appearance but offer weaker protection. Mid-frequency updates achieve comparable suppression to low-frequency updates with less distortion. We therefore propose MidGuard, a frequency-aware adversarial protection method for personalized diffusion models. It fuses Gaussian band-pass-filtered mid-frequency gradients with raw gradients to retain complementary signals, while progressively decays the contribution of mid-frequency guidance within each inner optimization loop to enable less spectrally constrained refinement. Extensive experiments demonstrate stronger overall identity protection than representative adversarial defenses while maintaining competitive visual fidelity. Code is available at [https://anonymous.4open.science/r/luck](https://anonymous.4open.science/r/luck).

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Diffusion Models, Adversarial Protection, Identity Privacy, Frequency-Aware Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_33VWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MidGuard, a frequency-aware adversarial protection method designed to prevent unauthorized identity personalization of diffusion models. The authors investigate the effects of different frequency components of adversarial gradients on identity protection and visual fidelity. Their analysis suggests that mid-frequency components provide a favorable trade-off between disrupting identity learning and preserving the visual quality of protected images.

Critical strengths
  1. The paper is clear and easy to follow.
  • The paper is generally well organized and clearly written. The motivation, frequency analysis, method formulation and experimental evaluation follow a logical progression.
  1. The solution is interesting.
  • The frequency-based analysis offers useful empirical observations about the trade-off between identity protection and visual fidelity.
Critical weaknesses
  1. Insufficient evidence supporting the proposed frequency-aware mechanism.
  • The central claim that mid-frequency gradients provide a superior protection–fidelity trade-off is not fully supported by the ablation results. In Table 3, mid-frequency filtering alone consistently underperforms Raw PGD on all protection metrics, while fixed fusion provides only limited improvements.
  • The theoretical analysis establishes a local sufficient condition rather than explaining why mid-frequency gradients should generally be preferred. The paper also lacks an analysis of whether the intended spectral properties survive the sign and projection operations used in practical PGD. These limitations make it difficult to isolate the actual source of the improvements and assess the novelty of the proposed mechanism.
  1. Insufficient experimental reliability and generalization analysis.
  • The reported results exhibit substantial variation across different ablation studies. For instance, the Face Similarity of MidGuard varies from 0.3154 in Table 3 to 0.2378 in Tables 14–15 and 0.0382 in Table 16.
  • The main quantitative evaluation relies on a single identity-conditioned prompt, and the evaluation does not cover several realistic attacker adaptations, such as combining protected and unprotected images or changing personalization duration. These limitations weaken the evidence for practical robustness.
  1. The frequency domain design is not particularly impressive and lacks innovation, it feels more like a trick.
  2. Mismatch between the motivation and the proposed frequency-filtering mechanism.
  • The paper identifies excessive perturbation allocation to background regions as a key limitation of conventional adversarial optimization. However, the proposed global Gaussian band-pass filter operates solely in the frequency domain without explicitly distinguishing identity-relevant facial regions from the background. Since both regions contain mid-frequency information, emphasizing these components does not necessarily redirect perturbations toward identity-related features. Consequently, it remains unclear how the proposed method directly addresses the spatial inefficiency identified in the motivation.
Questions for authors

See weakness and solve my concerns.

Reviewer_wGcsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes the performance of image protection methods such as Anti-DreamBooth across different frequency ranges and finds that the mid-frequency range performs best. Based on this observation, the authors propose MidGuard. The proposed method shows better performance than baseline methods across many metrics.

Critical strengths

The proposed MidGuard can be applied to existing methods in a versatile manner and is easy to utilize and straightforward.

Practically, it can meaningfully improve the performance of existing methods. It outperforms existing methods on multiple metrics in the face domain. The fidelity of protected images is also better than that of existing methods.

Critical weaknesses

Although the method is very simple and effective, I have several concerns and suggestions.

  1. Although the paper includes experiments on robustness and model transfer, the purification setting is limited to simple image preprocessing (e.g. Figure 4). Recently, various purification methods such as IMPRESS, Noisy Upscaling, and img2img-based approaches have been proposed, and I think results against these methods should also be reported [1, 2].

  2. (Minor) For model transfer, the performance on SDXL is not very strong. I understand that many existing protection methods mainly focus on SD 1.x or 2.x, and there have been bunch of reports that transfer to SDXL is difficult, and apparently this method appears to have the same limitation. I think some analysis of this point would be useful. Although MidGuard achieves the best performance on many metrics in Table 5 and the authors describe this as "strong protection", in protection research, the performance can vary considerably depending on the seed randomness of LoRA personalization. Therefore, I am not fully convinced that the improvement here (at least -> SDXL) is significant.

  3. In the face domain, various personalization-oriented methods such as MetaCloak, CAAT, and ACE have been proposed, but the comparison with these methods is limited. Instead, most of the compared baselines seem to be editing-focused methods, which I find somewhat strange.

  4. My biggest concern is that there have already been many studies analyzing similar aspects from the methodological perspective. Several papers such as DCT-Shield, DDAP, and AntiPure have discussed the importance of the frequency domain for image protection. Although the setting is slightly different because it focuses on targeted attacks, FastProtect has also discussed frequency characteristics of target images. Therefore, I am not fully convinced that the main observation itself is very novel.

[1] Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI, ICLR25

[2] Off-The-Shelf Image-to-Image Models Are All You Need To Defeat Image Protection Schemes, SaTML26

[3] Towards Robust Defense against Customization via Protective Perturbation Resistant to Diffusion-based Purification, ICCV25

[4] DDAP: Dual-Domain Anti-Personalization against Text-to-Image Diffusion Models

[5] DCT-Shield: A Robust Frequency Domain Defense against Malicious Image Editing, ICCV25

Questions for authors

I wonder how well the method generalizes to SDXL and performs against more recent purification methods. In addition, I think the paper would benefit from more discussion with recent frequency-aware protection methods.

Reviewer_LL6FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies protection of personal images against unauthorized identity learning through diffusion-model personalization. It proposes MidGuard, which filters adversarial diffusion gradients toward a Gaussian mid-frequency band, fuses them with raw gradients, and relaxes the frequency guidance linearly within each PGD loop. The method is evaluated across a wide range of settings and achieves strong results compared to several other baselines.

Critical strengths
  1. The paper provides a useful empirical observation and turns it into an actionable method. Fig. 2 compares low-, mid-, and high-frequency gradient updates under a common perturbation budget, showing the proposed mid-frequency region achieves a substantially better protection-fidelity compromise than the two extremes. Eq. 2–7 then translate this observation into a simple, understandable gradient reweighting procedure.

  2. The empirical evaluation is broad, and the main results are competitive across multiple metrics. The experiments cover two datasets, three diffusion backbones, two personalization methods, six preprocessing transformations, two source-to-target transfers, and necessary ablations. MidGuard outperforms six other adversarial defenses in terms of both protection effectiveness and image quality retention.

Critical weaknesses
  1. MidGuard explicitly emphasizes mid-frequency adversarial gradients, which may make it more vulnerable to targeted frequency-selective countermeasures than existing defenses. For instance, could a simple mid-frequency band-stop filter weaken or even eliminate its protective effect while preserving the visual quality? MidGuard's robustness against frequency manipulations and its reliability in practical deployment remain unclear.

  2. The same MidGuard configuration yields very different results across tables, despite the authors stating that the ablation studies use the same CelebA-HQ setup as the main experiments. For example, the reported Face Slim.\text{Face Slim.} varies in Tab. 1 (0.31100.3110), Tab. 3 (0.31540.3154, Complete MidGuard), Tab. 15 (0.23780.2378, c=0.45c = 0.45), and Tab. 16 (0.03820.0382, Reverse). If these variations are caused by random seeds, such prominent discrepancies raise concerns about the stability and trustworthiness of the reported values.

Questions for authors
  1. Could the authors provide a spectral analysis of the update directions after the sign\mathrm{sign} operation? Has the sufficient condition (Rmid>max⁡(Rlow,Rhigh)R_{\mathrm{mid}} > \max(R_{\mathrm{low}}, R_{\mathrm{high}})) supposed in A.2 been numerically verified? Is there evidence that MidGuard improves perturbation allocation over Raw PGD by face/background latent shifts?

  2. The paper cites DDAP (Yang et al., 2024) but omits it from the experimental comparisons. DDAP investigates spatial-domain and frequency-domain perturbation learning and combines both approaches to improve identity protection and visual fidelity. Can the authors clarify whether DDAP is incompatible with the SD1.5 DreamBooth evaluation protocol used in this paper?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.