acceptodds
Under review as a conference paper at ICLR 2027

DiTCloak: Protecting Facial Privacy Against Personalized Text-to-Image Diffusion Transformers via Unlearnable Examples

Abstract

Diffusion Transformers (DiTs) increasingly serve as denoising backbones for modern text-to-image (T2I) models. Their adoption in subject personalization, however, enables publicly shared portraits to be learned and reproduced without permission. Unlearnable examples (UEs) seek to mitigate this by applying concealed perturbations before image release. Yet prior approaches largely inherit objectives developed for U-Net architectures, leaving Transformer block designs in DiTs largely unexplored. Accordingly, we present DiTCloak, which optimizes perturbations at both the latent diffusion and DiT structural levels. At the latent diffusion level, we study how guidance interacts across VAE encoding and diffusion prediction, revealing that guiding both toward a shared target pattern is more effective, thereby motivating our target-guided dual alignment objective. At the DiT structural level, token relation anchoring reshapes inter-token dependencies according to the target pattern, limiting the learning of subject-specific local structure. Meanwhile, we disrupt multi-head attention within Transformer blocks to impair visual information aggregation in DiTs, forming the multi-head attention reorientation objective. Together, these objectives are optimized by alternating surrogate adaptation with projected gradient descent (PGD). Experiments on CelebA-HQ and VGGFace2 across diverse personalization schemes and DiT architectures demonstrate that DiTCloak outperforms nine state-of-the-art defenses in protection performance. Our code is available at [https://anonymous.4open.science/r/DiTCloak-7E67](https://anonymous.4open.science/r/DiTCloak-7E67).

Then back it, or bet against it.

Related papers

Open the market on this paper to see 7 more related papers.