CAPure: Rethinking Purification-Resistant Personalization Defenses via Identity-Aware Diffusion Purification

Text-to-image diffusion models such as Stable Diffusion can learn a person's visual features from a few reference images, raising the risk of unauthorized personalized generation. Existing defenses against unauthorized personalization add protective perturbations to reference images. Recent purification-resistant protective perturbations further reduce the separability of perturbations from meaningful image content, making it difficult for existing purification methods based primarily on natural image priors to reliably neutralize their protective effects. We seek to bypass such protection to test whether existing purification paradigms overestimate its robustness. To this end, we analyze how protective perturbations disrupt downstream identity learning to identify a more direct target for restoring identity learnability. Experiments show that protective perturbations substantially suppress the cross-attention responses of generic person-related tokens within person regions. This observation motivates using these responses as a guidance signal for restoring identity learnability. Based on this finding, we propose CAPure, a purification method that directly restores identity learnability. It uses the cross-attention responses of person-related tokens to guide diffusion purification, recovering suppressed person-semantic responses and facilitating subsequent identity association, while constraining stepwise clean-image predictions to limit content distortion and identity drift. The method operates solely at the input purification stage and requires no modification to the downstream personalization pipeline. Experiments on CelebA-HQ and VGGFace2 show that CAPure more effectively restores identity learnability against various purification-resistant protection methods, revealing remaining vulnerabilities in existing purification-resistant protective perturbations.