acceptodds
Withdrawn from ICLR 2027

CAPure: Rethinking Purification-Resistant Personalization Defenses via Identity-Aware Diffusion Purification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-to-image diffusion models such as Stable Diffusion can learn a person's visual features from a few reference images, raising the risk of unauthorized personalized generation. Existing defenses against unauthorized personalization add protective perturbations to reference images. Recent purification-resistant protective perturbations further reduce the separability of perturbations from meaningful image content, making it difficult for existing purification methods based primarily on natural image priors to reliably neutralize their protective effects. We seek to bypass such protection to test whether existing purification paradigms overestimate its robustness. To this end, we analyze how protective perturbations disrupt downstream identity learning to identify a more direct target for restoring identity learnability. Experiments show that protective perturbations substantially suppress the cross-attention responses of generic person-related tokens within person regions. This observation motivates using these responses as a guidance signal for restoring identity learnability. Based on this finding, we propose CAPure, a purification method that directly restores identity learnability. It uses the cross-attention responses of person-related tokens to guide diffusion purification, recovering suppressed person-semantic responses and facilitating subsequent identity association, while constraining stepwise clean-image predictions to limit content distortion and identity drift. The method operates solely at the input purification stage and requires no modification to the downstream personalization pipeline. Experiments on CelebA-HQ and VGGFace2 show that CAPure more effectively restores identity learnability against various purification-resistant protection methods, revealing remaining vulnerabilities in existing purification-resistant protective perturbations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Personalized Diffusion Models, Protective Perturbations, Diffusion Purification, Adversarial Robustness, Generative AI Security

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jo3hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a purification method for attacking current purification-resistant personalization defenses. Specially, CAPure focuses person-related tokens for diffusion purification, which can recover suppressed person-semantic responses and facilitates subsequent identity associations.

Critical strengths
  1. This paper considers a new attack for purification-resistant personalisation defences.
  2. The writing is well and easy to follow.
Critical weaknesses
  1. The finding is not surprising because the method of protecting portraits will undoubtedly focus on the face area. Besides, the proposed method largely combines existing cross-attention guidance and reconstruction-based purification. These are existing techniques in personalized diffusion models, e.g., DisDiff, CoPSAM, Attend-and-Excite.

  2. The ablation results indicate that reconstruction guidance accounts for most of the identity recovery, whereas the proposed attention guidance provides only modest additional gains. This weakens the claimed importance of the central identity-aware design.

  3. The proposed method has extensive iterative denoising and additional gradient computations, yet the paper provides no runtime or computational cost. It remains unclear whether its relatively modest gains over existing purifiers justify the additional cost.

  4. The evaluation is concentrated on DreamBooth with Stable Diffusion v1.5. The authors should add generation experiments on other personalization paradigms (such as LoRA) and modern diffusion architectures.

  5. In Table1, all defenses do not seem to perform well, such as MetaCloack and HF-ADB under purification. The authors need to illustrate this. Otherwise, the experiments may have some mistakes. In addition, there are many errors in the red and blue markings of Table 1. For example, in the first column, 29.23 is not the best, but it is marked in red.

  6. The effect of the method seems to be very weak, especially considering that the focus of this article is on human identity, so ISM should be more important, but it seems that the improvement is very small, such as 0.678 v.s. 0.671, 0.670 v.s. 0.087. This makes the problem described in the article possibly not valid or not so important, because the original DiffPure and other methods will also automatically focus on the face area during the optimization process.

  7. The corresponding defense needs to be considered and analyzed, as the purification considered in the article is a kind of attack.

Questions for authors
  1. What is the key technical novelty of the proposed method, especially considering the limited contribution of attention guidance in the ablation studies?

  2. Why the relatively marginal improvements in identity preservation given the additional computational overhead, and can the effectiveness be demonstrated on LoRA-based personalization and more recent diffusion architectures?

  3. Can the authors explain the unexpectedly weak performance of existing defenses in Table 1?

Reviewer_ict7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that cross-attention to the person token in Stable Diffusion can be leveraged to steer the purification of protected face images. The proposed method CAPure combines this attention guidance with a reconstruction term that keeps each denoising step close to the protected input, effectively weakening the protective perturbations that prevent personalization models from learning the person’s identity. The authors showed that the approach achieved the highest ArcFace ISM among the compared purification methods on CelebA-HQ and VGGFace2, under five protection methods.

Critical strengths
  • The paper identifies concrete evidence that protective perturbations suppress cross-attention to the person token and demonstrates that existing purifiers (GrIDPure, UDAP) can not not restore it.
  • The method is training-free, needs no clean counterpart of the protected image, and allows easy integration into the downstream personalization pipeline.
  • The method achieves the highest ISM in every tested setting, as shown throughout the results section.
Critical weaknesses

W1. The attention analysis is limited despite being one of the main motivations of the paper. Only the person token is examined, without investigation of other person-related tokens (e.g., face, man) or unrelated tokens (e.g., tree, for controls).

W2. The performance improvements are small. ISM gains over the best baseline are about 0.01 (Table 1). A small gain is also reported in Table 4, which might indicate that the attention mechanism does not seem to drive the gain over reconstruction

W3. The experimental scope is limited. For example, the test set is also limited to 25 identities per dataset. In addition, the cited defenses (CAAT, DisDiff, CoPSAM, AntiPure) are not tested, and only DreamBooth was studied as the editing approach, although Fig. 1 mentions about LoRA and Textual Inversion. Without a wider evaluation range, many claims in the paper are not sufficiently supported.

Minor Weaknesses:

  • No hyperparameter ablation for Eq. 2–3.
  • MetaCloak notes BRISQUE can rate poisoned outputs better than clean ones, which seemed to happen in this work. The paper already acknowledges this limitation (Sec. 4.1), yet still claims fewer artifacts (Sec. 4.2).
  • Purified images were not assessed directly, and most of the evaluation was based on downstream generations.
  • Missing details: chosen hyperparameters (tau, noise level) and how they were chosen.
  • Typos: “Lager” (Fig. 9); “CApure” (Figs. 3–4).
Questions for authors
  1. Why is the sks token included in the purification text condition? What’s the impact of removing it/ replacing it with other tokens?
  2. How much do calibration identities overlap with evaluation identities?
  3. In loose terms, could a defense exploit on this finding and preserves person-token attention and is able to evade CAPure?
Reviewer_S6VUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to bypass protective perturbations in personal photos, enabling downstream personalized diffusion models to relearn the target person’s identity. The authors propose CAPure, an input purification method that uses cross-attention guidance from person tokens to restore person-related semantic responses and stepwise reconstruction constraints to preserve appearance, without modifying the downstream personalization pipeline. Experiments show that CAPure improves the identity similarity of downstream generated faces under the evaluated protection methods.

Critical strengths
  1. The paper examines whether purification-resistant protection methods can still be bypassed, addressing a practically relevant problem. The authors shift the purification objective from restoring image appearance to restoring downstream models’ ability to learn identity, offering a meaningful perspective on evaluating protective perturbations.
Critical weaknesses
  1. Generalization across generative models and personalization pipelines remains insufficiently evaluated. The main experiments use SD v1.5 and DreamBooth, while the supplementary SD v2.1 experiments include only two protection methods. Since the guidance signal comes from specific attention layers in SD v1.5, evaluation on generative models with more distinct architectures and alternative personalization methods would help establish the method’s applicability.

  2. The method has not been systematically evaluated under common image transformations, such as compression, resizing, cropping, and blurring. Real-world photos often undergo platform compression or resolution changes. Evaluating whether these transformations affect attention guidance and subsequent identity recovery would strengthen the empirical analysis.

  3. The implementation details are incomplete, limiting reproducibility. Although the paper provides the main equations and some parameter values, it does not sufficiently specify the noise strength, attention thresholds, or how the guidance weights vary across denoising steps.

  4. The computational cost and the trade-off between performance and efficiency remain unclear. CAPure requires 200 denoising steps per image, along with additional gradient computations for attention guidance and reconstruction, but the paper reports neither processing time nor GPU memory usage and provides no comparisons under matched computational budgets.

Questions for authors

No more questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.