acceptodds
Withdrawn from ICLR 2027

DiffUnlearn: Selectively Unlearnable Examples via Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Images shared online can be used for model training without their owners' consent. To protect images from such unauthorized training, unlearnable examples add small perturbations before release to impair the generalization of models trained on the protected images. However, data owners may wish to protect only a target concept while retaining the training utility of non-target concepts within the same image. This motivates selective protection of concepts within the same image, a requirement that remains underexplored in existing approaches. To address this gap, we formulate this goal as selective unlearnability and propose DIFFUNLEARN, a diffusion-based framework for protecting a text-specified target concept. Our protection objective combines image-text alignment, target attention suppression, and visual preservation. To optimize this objective, we use a pretrained diffusion model to perform text-guided optimization of intermediate latents obtained through DDIM inversion of the input images. First, we alternate vision-language surrogate training with latent optimization, using image-text alignment during latent updates to encourage target-correlated shortcuts that reduce reliance on generalizable target features. Second, to provide spatial guidance, we suppress the diffusion model's target attention responses in regions associated with the target concept. Finally, the visual preservation loss constrains image distortion under a fixed perturbation budget. Experiments on multilabel classification show that DIFFUNLEARN suppresses learning of the selected target while largely preserving aggregate non-target training utility. We additionally design transfer evaluations across unseen models and downstream tasks using the same protected images without task-specific reoptimization. Code is available at https://anonymous.4open.science/r/9F6B/

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
unlearnable examples, selective unlearnability, diffusion models, data protection, vision-language learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VnYWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DiffUnlearn, a method to generate perturbed training images to suppress learning of user-designated semantic categories while preserving non-target utility. It optimizes diffusion latents using image–text alignment, attention suppression, and visual fidelity losses under a pixel-level perturbation budget. Its main contribution is a diffusion-based approach to category-selective protection through unlearnable examples. The authors conduct experiments across multiple datasets and models, and the results support the method’s ability.

Critical strengths
  1. The method provides a coherent approach to selectively inhibiting target-category learning while maintaining visual fidelity, giving the work methodological merit. The overall framework and individual components are rationally designed and logically clear.

  2. The experiments evaluate both target-category learning suppression and non-target utility preservation across multiple datasets and models, with supporting ablation studies. This directly tests the selective protection capability and strengthens the evidence for the method’s effectiveness.

Critical weaknesses
  1. I find the specific task setting novel, and the method achieves promising results. However, its practical value remains my main concern. Users would typically want to protect a particular person, character, or IP rather than a broad semantic category. Could the authors clarify what concrete user need is addressed by making a designated category difficult to learn, and how this translates into a practical protection benefit?

  2. The main components have precedents in related work, so I find the methodological novelty somewhat limited. The overall design appears to be an effective combination of existing ideas, but lacks deeper insights and analysis.

Questions for authors
  1. As discussed in the critical weaknesses, the authors should first explain how the proposed framework can be applied in practical scenarios and what its key new design is. What evidence shows that its advantages go beyond a simple combination of existing components?

  2. If the trainer (attacker) introduces additional clean images of the same target category from their own sources, could this undermine the claimed protection? Does the protection depend on the proportion of clean data?

  3. The anonymous code currently appears insufficient to reproduce the results, and its settings do not fully match those reported in the paper.

Reviewer_2zMMOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a diffusion-based framework, DiffUnlearn, to generate unlearnable examples that selectively protect objects/concepts in images. It first maps each image to an intermediate diffusion latent through DDIM inversion and derives a target mask from text-conditioned attention. It then alternates surrogate training with latent updates that combine image-text alignment, target attention suppression, and visual preservation. The decoded images are clipped to a fixed perturbation budget to reduce target learning while retaining non-target utility.

Critical strengths
  • Selective protection of certain concepts is a merit for constructing fine-grained level unlearnable features.
Critical weaknesses
  • The technical advance over existing methods is limited. DiffUnlearn combines diffusion latent editing with alignment-based unlearnable example generation. Although selective protection is useful, the main text does not show how much target attention suppression adds beyond visual preservation alone. A direct comparison without attention suppression would clarify the contribution of this component.

  • The practical protection remains insufficiently evaluated. The main text reports transfer across models and tasks, but does not test preprocessing intended to remove protective perturbations before training. An unauthorized trainer may not use the released images unchanged, but instead they may apply augmentations or purification to the images for pre-processing. Testing a simple intervention such as JPEG compression would help establish whether selective protection survives this setting.

Questions for authors
  1. How do target and non-target performance change when target attention suppression is removed while the other losses are retained?

  2. Does selective protection remain effective when the training images are JPEG-compressed, using identically compressed clean images as the reference?

Reviewer_XipjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a novel type of unlearnable examples that can selectively protect certain concepts in images from being learned by downstream models. The authors assert that the method can specifically target a text-specified concept and render it unlearnable while keeping other concepts learnable. They employ a pretrained diffusion model to modify the diffusion latent and a separate CLIP surrogate to establish a shortcut, prompting the downstream model to learn the shortcut rather than the actual target concept in the original image. The procedure also reduces diffusion attention toward the target and maintains visual appearance. The authors subsequently perform extensive experiments to demonstrate that this protection can generalize across downstream models, with models trained on the protected images exhibiting poorer performance on unseen images containing the same target concept.

Critical strengths
  1. The authors propose concept-selective unlearnability, aiming to protect a text-specified concept from model training while preserving the training utility of other concepts within the same images.
  2. The authors design a detailed framework combining diffusion-based editing with vision–language guidance. Experiments across several architectures and tasks demonstrate target suppression with largely retained non-target performance.
  3. The experiments provide encouraging evidence for the effectiveness and transferability of the proposed method. In particular, Table 4 shows that DiffUnlearn reduces target AP by approximately 33–38 percentage points compared with clean training across four classification architectures, while maintaining non-target mAP within 0.75 percentage points of the clean baseline. Furthermore, the component ablations in Table 7 demonstrate the complementary contributions of alignment, attention suppression, and visual preservation, providing empirical support for the overall framework design.
Critical weaknesses
  1. Limited practical relevance of the evaluated concepts: The method relies on pretrained models that are already familiar with the common MS COCO categories used in the evaluation. If these models have already learned the concepts well, the value of protecting them from learning is questionable. The paper does not test rare or newly created concepts, where selective protection could have greater practical value and the required pretrained semantic knowledge may be unavailable. Also, it is unclear how much benefit there is in preserving non-target concepts when similar information can be learned from other images.
  2. Missing comparison with a simple masked baseline: The paper does not compare against a straightforward pipeline that first localizes the target concept and then applies an existing UE method only within that region, leaving the advantage of the proposed construction unclear.
Questions for authors
  1. Comparison with masked existing methods: Could a simpler approach—just finding the target area and applying an existing protection method there—work just as well(or even better since they apply less noise on non-target concepts)? Please compare your method with existing ones using the same data, area detection, and noise limits.
  2. Practical targets and clean-data exposure: What real problem is this method trying to solve? Common objects like dogs or people already have plenty of clean images online, so protecting them might not be very useful. Can the method protect new or unique things, like a rare concept, that the pre-trained models didn't perform well? Also, how well does it work if the model sees both protected and clean images of the same thing? Please test this by changing how many protected images are in the training set. Just keeping clean non-target images isn't enough; adding clean target images from another source would help show if it really works in practice.
  3. Downstream training coverage: Table 5 only tests classifiers using frozen features from MLLMs, which doesn't prove it works during actual fine-tuning of MLLM. Could you test it during real MLLM fine-tuning or diffusion model training to see if it still blocks the target and keeps other features useful? Those are the models that are likely to be trained on the protected images.
  4. Evidence of shortcut reliance: The tests show that the alignment loss helps with protection, but they don't prove that the downstream models are actually relying on the "shortcuts", while the shortcut is the central design of the entire work. Can you provide direct proof to show that the shortcut is actually used by the downstream models? 5.Real application: In what real-world scenarios would image owners want to protect one concept while allowing models to learn other concepts from the same image?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.