acceptodds
Under review as a conference paper at ICLR 2027

PuriShield: Robust Defense against Unauthorized Diffusion Customization via Structured Protective Perturbations

Abstract

Latent diffusion models have raised severe security concerns regarding unauthorized customization. Existing anti-customization defenses protect images by injecting subtle protective perturbations, yet such unstructured perturbations suffer from poor robustness against Diffusion-based Purification (DP). To tackle this limitation, this paper proposes , a novel anti-customization method that trains a Robust Protective Perturbation Generator (RPPG) to adaptively produce structured protective perturbations. Specifically, incorporates three core designs: pattern density-guided generation to yield structured perturbations, latent distribution contrastive loss to refine protective perturbations, and embedded dual-head modulation to enable high-quality perturbation synthesis. Extensive experimental results demonstrate the advantages of : ① Excellent effectiveness. It effectively defends against 5 unauthorized customization methods prior to purification. ② Strong robustness. Across 13 purification methods, it attains optimal robustness and significantly surpasses baselines against UDAP, a DP approach tailored to counter anti-customization defenses. ③ Great generalizability. It performs well in cross-task and cross-domain scenarios. ④ High efficiency. Training the RPPG takes only 31.6 minutes, with a mere 5.8 ms inference latency.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Diffusion Model, Protective Perturbation, Anti-Customization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GvNkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PuriShield to protect images from unauthorized diffusion customization, including customization after image purification. It trains a perturbation generator with sinusoidal pattern conditions and dual-head modulation. A latent distribution loss moves protected images toward the current perturbations and away from clean images. The method also uses alternating training, a perceptual loss, and random block masking. Experiments cover several customization methods, purification methods, datasets, and model architectures. The results suggest that PuriShield offers a useful balance between protection strength, purification robustness, and inference efficiency.

Critical strengths
  1. The paper reports promising protection after purification. The evaluation covers diverse datasets, customization methods, and purification attacks.
  2. The generator enables fast protection in a single forward pass. The browser extension adds practical value, and the ablations help assess the main design choices.
Critical weaknesses
  1. The novelty needs a clearer comparison with closely related work. BlurGuard already uses frequency-aware perturbations to improve robustness against purification. PatchOut randomly masks perturbation patches during optimization, while Anti-Reference uses a learned noise encoder for fast protection. These methods provide close precedents for several components of PuriShield. The proposed combination may still offer a useful contribution, but the paper does not clearly separate its new elements from these established ideas.
  2. The evidence for the proposed manifold-based explanation is limited. Table 29 uses VAE round-trip consistency to support the claim that protected images remain close to the natural-image manifold. However, consistency under the VAE mapping does not directly establish proximity to natural images, and the table does not provide a clean-image reference. Table 30 also reports “Riemannian distances” without clearly defining the metric or its computation.
  3. Whether training and evaluation identities are disjoint? Although adaptive attacks such as PDAP are included, their filtering settings and attack strengths are not sufficiently specified for reproduction. In addition, the reported ISM before purification differs between Table 1 (0.098) and Table 26 (0.138), without a clear explanation of the setup differences.
Questions for authors

Please refer to the Weaknesses.

2 comments
Reviewer_GvNkOpenReview ↗

Comment on Rebuttal

I thank the authors for reporting additional results regarding my concerns, so I maintain my positive assessment and my rating.

Thank you for your constructive feedback and continued positive assessment of our work.

We appreciate your time and effort during the review and discussion.

Reviewer_6pYxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PuriShield aims to prevent unauthorized diffusion customization using protective perturbations that remain effective after purification. Existing defenses generate unstructured perturbations that are easily removed by diffusion-based purification (DP) methods. To address this, PuriShield trains a robust protective perturbation generator that generates structured perturbations by combining pattern-density guidance (yielding a structured prior), latent distribution contrastive loss, and a dual-head modulation mechanism. The authors claim that this method outperforms existing baselines in effectiveness, robustness against adversarial purification, and cross-task/domain generalization.

Critical strengths

[S1] The paper addresses an important practical limitation and provides broad empirical evaluation. Protective perturbations have limited utility if routine preprocessing removes their effect. PuriShield directly targets this limitation and evaluates transformation-based and diffusion-based purification across multiple customization methods. The appendix additionally considers cross-model transfer, latent-variance manipulation, and pattern-density-aware purification. The robustness of this method is good.

[S2] The amortized generation approach has practical value. A trained generator avoids running iterative optimization for every protected image. The dynamic perturbation target and spatial modulation offer a plausible approach to combining protection strength with resistance to purification.

Critical weaknesses

[W1] Unresolved numerical inconsistencies and incomplete experimental documentation weaken confidence in the reported results. Several results require verification. Tables 1 and 26 report different default pre-purification results for PuriShield without explaining the difference. Tables 12 and 14 contain identical PuriShield IQS values under three purification methods and the same UDAP FID, despite evaluating different downstream methods. Table 20 also reports a PSNR of 33.656 dB for Anti-Diff at an L-infinity budget of 4/255. Under the stated pixel-space constraint and standard PSNR computation on the same images, PSNR should be at least approximately 36.09 dB. These observations may need to be reconciled before the comparisons can be assessed confidently.

[W2] The robustness evaluation does not yet establish reliable protection under a sufficiently specified attacker model. The paper evaluates many transformations and purification methods, and Table 28 includes mechanism-aware attacks. However, important parameters and tuning procedures are insufficiently documented, including resizing configurations, purification strengths, PDAP settings, and whether LightShed and CAT were specifically adapted to PuriShield. Most purification tables also omit the corresponding clean-image-plus-purification control, making it difficult to distinguish residual protection from degradation caused by purification itself.

[W3] The artistic-style evaluation does not directly establish that style imitation has been prevented. For example, Table 14 reports an ISM of 0.658 under LoRA with Noisy Upscaling. FID and image-quality metrics cannot determine whether generated outputs remain useful for identity or style imitation. IQS and LIQE primarily assess output quality, while standard FID, LPIPS, and MS-SSIM do not isolate stylistic fidelity from content differences or visual corruption. In particular, a generated image can depict different content while successfully reproducing the target artist’s style, or retain that style despite reduced visual quality.

[W4] The proposed natural-manifold explanation and dynamic-target argument are not adequately supported. Sinusoidal conditioning imposes a spatial prior, but does not establish that the resulting perturbations lie on the natural-image manifold. The t-SNE visualization and VAE round-trip error are indirect evidence, and the reported Riemannian distance lacks a sufficiently detailed definition. The ablations demonstrate that components help, but do not isolate manifold alignment as the cause of purification resistance. The dynamic-target explanation also needs clarification: bounding the additive perturbation delta does not imply that delta, interpreted as a target image, lies within the feasible neighborhood of the original image x. The triplet objective constrains relative distances rather than guaranteeing absolute proximity to its positive target. Furthermore, Equation (3) and Algorithm 1 update the surrogate diffusion model using delta rather than x + delta, leaving its correspondence to actual unauthorized customization unclear. These gaps matter because the manifold and dynamic-target interpretations are central to the claimed contribution.

[W5] The practical advantage and scope of generalization remain insufficiently established. The protection gains are not evaluated at comparable perceptual quality. At the default budget, Table 20 reports LPIPS values of 0.465 for PuriShield and 0.133 for BlurGuard. Thus, matching the L-infinity budget does not establish that the methods impose comparable visual costs. A perceptual-quality-matched comparison or protection–distortion curve is needed to assess the practical improvement over the closest defenses. The generalization evidence also does not fully support the artistic-protection use case. Tables 16–18 evaluate cross-model transfer on faces, while the artistic-style experiments evaluate cross-domain transfer without establishing equivalent robustness across target model families. These two results do not jointly demonstrate that an artist can use the SD 1.5-based protector against an unknown style-imitation model. This matters because users cannot choose the model an unauthorized party will employ.

Questions for authors

[Q1] Result verification: Can you reconcile the PuriShield results in Tables 1 and 26, verify the repeated entries in Tables 12 and 14, and explain the budget–PSNR discrepancy in Table 20? Please identify any differences in preprocessing, evaluation samples, or random seeds.

[Q2] Robustness protocol: Are training, hyperparameter selection, and evaluation separated by identity? What exact compression, resizing, and denoising parameters were used? Representative strength sweeps with corresponding clean-image controls would materially affect my assessment.

[Q3] Adaptive evaluation: Were LightShed and CAT trained or tuned specifically against PuriShield? What parameters and tuning budget were used for PDAP? Does the assumed attacker know only the pattern-density parameter, or also have access to the trained generator?

[Q4] Protection versus distortion: How is ISM handled when face detection fails? Can the authors provide a direct identity-success evaluation and a perceptual-quality-matched comparison against a strong baseline? These would clarify whether the method prevents useful customization rather than primarily lowering average output quality. The authors should complement the existing metrics with a validated style-sensitive evaluation.

[Q5] Mechanism and training: Is the surrogate diffusion model actually updated using delta, as written, or the protected image x + delta? Please clarify the claim that the dynamic target lies within the perturbation budget and provide the definition and implementation of the Riemannian metric.

[Q6] Generalization scope: The cross-model experiments concern faces, whereas the artistic-style experiments do not establish equivalent transfer across model families. Is there evidence for style protection against an unseen model family using the frozen SD 1.5-based protector? If not, the claims should distinguish cross-model identity protection from cross-domain artistic protection.

Reviewer_ED6MOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses a known weakness of anti-customization defenses that protective perturbations optimized per image with PGD are largely erased by diffusion-based purification. This paper propose PuriShield which trains a feed-forward generator that builds each perturbation from an image-dependent amplitude map and a learned mixture of fixed sinusoidal pattern-density embeddings, under budget of 8/255. The authors argue that these structured perturbations stay close to the natural image manifold and therefore survive purification. The generator is trained in alternation with DreamBooth fine-tuning, using a contrastive loss in the VAE latent space that pulls the protected image toward the perturbation and away from the clean image, a fine-tuning loss on the perturbation's latent, a perceptual loss on UNet features and random block masking. The method requires 5.8 ms of inference per image.

Critical strengths
  1. The proposed statement is well motivated. Fragility under purification is one the reasons that protective perturbations are not commonly trusted in practice.

  2. The empirical study covers thirteen purifiers, five customization methods, cross-model transfer and ten baselines, together with an adaptive Fourier-domain attack.

  3. The proposed defense is practical to deploy. Inference takes milliseconds per image, and on-device inference in a browser extension is a tangible step toward protection that ordinary users could use.

  4. Each component and the main hyperparameters are ablated, and the contrastive loss is compared with targeted, untargeted and fixed-target variants.

Critical weaknesses
  1. If I understood correctly, the perturbation's strength is set by clipping. The amplitude map A and the structural map W both take values in [−1, 1], while the final perturbation is clipped to 8/255. The network's raw output is therefore up to thirty times larger than the budget, and wherever it exceeds epsilon the clip, not the loss, determines the perturbation. The structural component is also image-independent by design where E_pd depends only on omega, so every protected image carries the same spatial frequencies. Both choices push the perturbation toward a high-energy, repeated pattern. This matters for visibility and for adaptive attacks, and the paper should report the fraction of clipped pixels and the L2 energy of δ against the baselines.

  2. The threat model ignores the defining property of a generator-based defense. An ideal defense is a single fixed function G that anyone can query without limit. The natural adversary therefore holds G and can generate a fair amount of pairs. The paper never states a threat model that includes this adversary, and none of the attacks it enables is evaluated where a purifier trained on such pairs, an estimate of δ obtained by re-applying G and subtracting it, or an inversion of x′ + G(x′).

  3. The comparison fixes the budget but not visibility, and PuriShield's perturbations are among the most visible. At ε = 8/255, PuriShield has LPIPS 0.465 and SSIM 0.807. The corresponding values are 0.133/0.961 for BlurGuard, 0.381/0.829 for Anti-Pure and 0.343/0.783 for Anti-Diff, and only ACE is worse on LPIPS. Protection must be compared at matched perceptual cost (LPIPS, SSIM or L2), ideally as protection-versus-perceptibility curves.

  4. The experiments lack statistical support, and I think the baseline protocol favors the method. No variance across seeds or identity subsets is reported, although each setting rests on 50 identities with 20 generated images each, and FID values above 300 computed on about 1,000 images are unstable. PuriShield's hyperparameters are tuned in ablations that use GrIDPure as the purifier, while the baselines run at their published defaults. MetaCloak, the closest prior work on purification-robust protection, is discussed in the related work but not compared.

Questions for authors
  1. What approach do you believe produces the protection? Do models fine-tuned on protected images reproduce the periodic pattern, which would indicate shortcut learning?
  2. What fraction of δ's pixels are clipped at ε, and how does δ's L2 energy compare with that of the baselines?
  3. How do all methods compare at matched LPIPS or SSIM rather than matched L?
  4. How robust is PuriShield to an attacker who holds G where a purifier trained on (x, x + G(x)) pairs, subtraction of G applied to the protected image, or inversion? Is LightShed retrained on PuriShield outputs?
  5. How do the baselines perform under PDAP?
  6. What are the metrics for clean images after each purification method, and the clean reference values for Table 18?
  7. What is the result of replacing E_pd with random conditioning at matched perceptibility, and what are the numerical values behind Figure 6?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.