DiffUnlearn: Selectively Unlearnable Examples via Diffusion Models

Images shared online can be used for model training without their owners' consent. To protect images from such unauthorized training, unlearnable examples add small perturbations before release to impair the generalization of models trained on the protected images. However, data owners may wish to protect only a target concept while retaining the training utility of non-target concepts within the same image. This motivates selective protection of concepts within the same image, a requirement that remains underexplored in existing approaches. To address this gap, we formulate this goal as selective unlearnability and propose DIFFUNLEARN, a diffusion-based framework for protecting a text-specified target concept. Our protection objective combines image-text alignment, target attention suppression, and visual preservation. To optimize this objective, we use a pretrained diffusion model to perform text-guided optimization of intermediate latents obtained through DDIM inversion of the input images. First, we alternate vision-language surrogate training with latent optimization, using image-text alignment during latent updates to encourage target-correlated shortcuts that reduce reliance on generalizable target features. Second, to provide spatial guidance, we suppress the diffusion model's target attention responses in regions associated with the target concept. Finally, the visual preservation loss constrains image distortion under a fixed perturbation budget. Experiments on multilabel classification show that DIFFUNLEARN suppresses learning of the selected target while largely preserving aggregate non-target training utility. We additionally design transfer evaluations across unseen models and downstream tasks using the same protected images without task-specific reoptimization. Code is available at https://anonymous.4open.science/r/9F6B/