acceptodds
Under review as a conference paper at ICLR 2027

GIFT: Gradient-aware Immunization of diffusion models against malicious Fine-Tuning with safe concepts retention

Abstract

Text-to-image diffusion models can learn semantic classes, like artistic styles or object categories, which model owners may need to remove to prevent harmful or unauthorized outputs. However, existing concept-erasure methods are brittle; adversaries can easily recover the erased concepts using simple fine-tuning attacks. This vulnerability occurs because erasure methods only increase the loss gap for the target concept but leave the underlying representational structure intact, which makes the concept easily "reachable" during fine-tuning. To address this, we introduce GIFT, an immunization technique designed to attack both the loss landscape and the hidden representation of a target concept, leading to a tamper-resistant model. GIFT pairs gradient ascent on the denoising loss with representation noising, a technique that pulls intermediate attention activations toward Gaussian noise to degrade the reachability of the target concept. To maintain general model utility on safe concepts, GIFT anchors internal representations to a frozen base model and utilizes gradient surgery to resolve optimization conflicts between the erasure and preservation updates. Extensive evaluations across art styles and object categories demonstrate that GIFT achieves superior initial erasure and remains significantly more robust against fine-tuning attacks compared to state-of-the-art baselines, all while maintaining fidelity on safe-concept generation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Diffusion Models, Concept Erasure, Model Immunization, AI Safety, Open-weight models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NXLuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GIFT combines gradient ascent on target denoising loss with representation noising to resist malicious fine-tuning. Training on safe data, reference activation matching, and asymmetric gradient projection preserve normal generation. Evaluations cover 14 styles and 10 object classes, SDXL, and multiple attack settings, showing improved resistance under the tested conditions with largely preserved SD1.4 generation quality.

Critical strengths
  1. The paper addresses a practically issue: erased concepts can be recovered via fine-tuning after weight release.
  2. The post-attack object ASR-1 drops from 0.450 for IMMA to 0.144 for GIFT, while performance on safe prompts remains close to that of the original SD1.4 model.
Critical weaknesses
  1. The method combines several established ideas. Although component ablations are provided, a controlled comparison against a simpler activation-suppression baseline is missing, leaving the added value of the full framework unclear.
  2. The representation-removal mechanism is mainly supported by output metrics and ablations. While object ASR is classifier-based, style metrics remain sensitive to image quality. Representation probing and style-recognition tests would further strengthen this claim.
  3. Main comparisons still focus on earlier methods, while recent baselines receive only limited discussion in the appendix. A broader multi-concept evaluation of compatible recent methods under a common fine-tuning attack protocol would be more informative.
  4. Stronger attacks mainly target Van Gogh, with some evaluated only qualitatively. Since both defender and attacker data are synthetic and from the same source, and the metrics are largely aggregate utility measures, practical generalization remains insufficiently tested.
Questions for authors

Although the paper addresses a meaningful problem and reports promising improvements, the added value of the proposed framework and the efficacy of its representation-removal mechanism remain insufficiently validated. In addition, the limited comparison with recent methods and the narrow coverage of stronger attacks weaken the broader claims.

1 comment

We thank the reviewer for the constructive review

1. Contribution & baseline (W1)

A controlled comparison against a simpler activation-suppression baseline is missing, leaving the added value of the full framework unclear.

(Novelty). Tamper-resistance for diffusion models is a relatively underexplored problem. Concept-erasure methods are undone by a few hundred fine-tuning steps, and existing immunization methods either still relearn the target under attack (IMMA, ResAlign) or lose general generation ability (Meta-Unlearning). Our main contribution is finding a recipe that works: which signals to apply (loss ascent, activation suppression), where to apply them (both self- and cross-attention, Table 5), and how to keep the rest of the model intact (safe activation anchoring and asymmetric gradient surgery, Table 4). No single component achieves this on its own, but together they resist the attack and preserve safe generation, at 6 minutes per concept.

(Simpler baseline). Because η\eta is sampled independently at every step, Eη∥z−η∥22=∥z∥22+d\mathbb{E}_{\eta}\lVert z-\eta\rVert_2^2 = \lVert z\rVert_2^2 + d, so Eq. (5) is equivalent to activation suppression in expectation. We rename the term representation suppression and revise its interpretation accordingly.

The simpler baseline is therefore the "Lnoise\mathcal{L}_{noise} only" row of Table 3. Suppression alone lets the attack restore the concept. Ascent alone leaves the concept reachable and lowers safe aesthetic score.

2. Target Erasure and Preservation (W2)

Style metrics remain sensitive to image quality. Representation probing and style-recognition tests would further strengthen this claim.

Target outputs: We do not claim to replace the target concept with another concept. GIFT is designed to suppress target generation by producing corrupted outputs, which will be captured by CLIP and AS scores. We explicitly show these outputs in Figs. 1–2. General safe generation is evaluated separately using MS-COCO metrics and qualitative examples.

3. Recent Baselines (W3)

Main comparisons still focus on earlier methods, while recent baselines receive only limited discussion in the appendix.

Appendix C.2 compares GIFT with ResAlign and Meta-Unlearning under the same attack protocol. ResAlign focuses on NSFW content and makes no claims for art styles or objects. Meta-Unlearning is closest to our setting. However, in our cassette-player experiment (Table 9), Meta-Unlearning with its published configuration lost general generation ability before any attack (COCO CLIP-T 0.121). GIFT took 6 minutes per concept, against 1.9 hours for ResAlign and 8.2 hours for Meta-Unlearning on the same GPU. We are happy to extend the comparison.

4. Stronger Attacks and Attack Data (W4)

Stronger attacks mainly target Van Gogh [...] both defender and attacker data are synthetic and from the same source.

The default 5k-step LoRA attack covers all 14 styles and 10 objects; the hyperparameter sweeps, DreamBooth and full fine-tuning are evaluated on Van Gogh only, which we state explicitly. Immunization and attack sets are disjoint, and the evaluation prompts differ from the immunization template, but both sets are generated by SD1.4. We agree that attacks with real images would better test generalization.

We hope these results and revisions address the reviewer's concerns, and would be glad to clarify further.

Reviewer_TwXHOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GIFT, a defense for diffusion models. It combines gradient ascent on the denoising loss of the target concept with a constraint on the attention representation, which suppresses recovery of the erased concept after harmful fine-tuning. To keep safe generation intact, the method adds a denoising prior, activation anchoring, and gradient projection. On artistic styles and object categories, the authors test several fine-tuning attacks with Stable Diffusion and SDXL, and they report good resistance to recovery along with good safe generation.

Critical strengths
  1. The problem has practical weight. It targets the risk that a deleted concept comes back after harmful fine-tuning.
  2. The experiments cover two concept families and two backbones, and they test several fine-tuning attacks.
  3. The ablations are fairly complete. They support the role of activation anchoring and gradient projection in safe generation.
Critical weaknesses
  1. The noise loss does not match the explanation. Equation (5) is a mean squared error against independent Gaussian noise. In expectation it pushes the activations toward zero, and it does not show that the representation distribution is matched to a Gaussian.
  2. The core theory has a gap in its derivation. Theorem 1 starts from the mutual information of the defended model and then infers bounded gradients along any later attack path. A necessary condition for that step is missing.
  3. Style evaluation cannot separate deletion from broken output. A lower image-text similarity and a lower aesthetic score may just mean the whole image turned into noise. They cannot on their own show that the style was removed.
  4. Validation under strong attacks is still narrow. The extended attacks center on Van Gogh, and a fixed template plus synthetic attack data is not enough to support broad adaptive robustness.
  5. The evidence for preserved safe capability is not fine-grained. The COCO average can hide damage to nearby concepts, and the safe generation metrics already drop on SDXL.
Questions for authors
  1. If the Gaussian noise target is replaced by a zero activation target, do the results stay the same?
  2. How does Theorem 1 constrain the gradient at the parameters of the fine-tuned model?
  3. Line 4 of Algorithm 1 has a different sign from Equation (3). Which one does the actual code use?
  4. Can a style classifier or human evaluation separate a vanished style from a damaged image?
  5. After the DreamBooth attack, do you test with the new identifier word that the attack learned?
  6. After one concept is erased, can nearby safe styles or objects still be generated normally?
  7. How many training and attack seeds were used for each result?
  8. Does the COCO evaluation use the 1,000 prompts from the main text or the 10,000 from the appendix?
1 comment

We thank the reviewer for the detailed and technically precise review.

1. Representation Noising (W1, W2, Q1, Q2)

We agree with both points. Because η\eta is sampled independently at every step,

Eη ∥z−η∥22=∥z∥22+d,\mathbb{E}_{\eta}\,\lVert z-\eta\rVert_2^2 = \lVert z\rVert_2^2 + d,

where dd is the number of activation coordinates. As clarified in our response to Reviewer 2Cka, Eq. (5) therefore penalizes the energy of target-concept activations; it does not match their distribution to a Gaussian. We also agree that a property of the defended initialization alone does not bound gradients along a later attack trajectory. We remove Theorem 1, rename the term representation suppression, and replace the argument in Appendix A with the following.

Consider one attention output projection. Let H⋆H_{\star} be its input features on target-concept inputs, W⋆W_{\star} the defended weights, Z⋆=W⋆H⋆Z_{\star} = W_{\star}H_{\star} the defended outputs, and Z0Z_0 the undefended model's outputs on the same inputs. For any update ΔW\Delta W of rank at most rr, such as a rank-rr LoRA,

\qquad Tail_r(M)=\Big(\sum_{j>r}\sigma_j(M)^2\Big)^{1/2}.$$ _Proof._ $\Delta W H_{\star}$ has rank at most $r$, so the first inequality follows from Eckart–Young–Mirsky. The second holds because $\mathrm{Tail}_r$ is 1-Lipschitz in the Frobenius norm. $\blacksquare$ ## 2. Deletion vs. Broken Output (W3, Q4) > Style evaluation cannot separate deletion from broken output. GIFT is designed to prevent generation of the target concept, not to replace it with another one; the noisy target outputs in Figures 1–2 are intended. Whether the rest of the model still works is measured separately, on MS-COCO and in the cross-concept evaluation below. We agree that a style classifier would complement these metrics. ## 3. Nearby Concepts (W5, Q6) > After one concept is erased, can nearby safe styles or objects still be generated normally? We ran a cross-concept evaluation for styles and objects. We evaluated the other 13 artists for each of the 14 style (14 * 13 pairs), and the same for objects (10 * 9). Erased-target values are from Tables 1 and 2. We used the same prompts to generate 100 images per concept per pair. | Concept family | Metric | SD1.4 | Non-target, after GIFT | Erased target, after GIFT | | -------------- | --------- | ----- | ---------------------- | ------------------------- | | Objects | ASR-1 | 0.73 | 0.60 | 0.000 | | Objects | ASR-5 | 0.90 | 0.81 | 0.000 | | Objects | CLIP-T | 0.25 | 0.25 | 0.162 | | Objects | Aesthetic | 5.2 | 5.02 | 4.020 | | Styles | CLIP-T | 0.29 | 0.23 | 0.161 | | Styles | Aesthetic | 5.27 | 4.30 | 3.891 | Objects are largely preserved, but styles show substantial collateral damage. One likely cause is the fixed immunization template (“A high quality painting in ⟨artist⟩ style”), whose shared tokens such as “painting” may be affected. We will further discuss this limitation. Collateral effects also arise with concept is filtering from the pre-training data [1]. That is also a known side effect of concept erasure [2]. A surgical isolation is an important future work. ## 4. Attack Protocol and Details (W4, Q3, Q5, Q8) **Q3 (Algorithm 1).** This is a typo. The code follows Eq. (3), with gradient ascent on the target denoising loss; we will fix Algorithm 1. **Q5 (DreamBooth).** Yes, we test with the new identifier word that the attack learned. **W4 (attack ablation).** We agree that broader attack evaluations would be valuable. However, our focus is on standard, low-cost fine-tuning attacks that already recover erased concepts from existing defenses, as shown in our experiments. **Q8 (COCO).** We use 1k prompts, as stated in the main text; the 10k in the appendix is a typo and will be fixed. We hope these results and revisions address the reviewer's concerns, and would be glad to clarify further. ------- [1] Creţu, Ana-Maria, et al. "Evaluating concept filtering defenses against child sexual abuse material generation by text-to-image models.". [2] Saha, Shaswati, et al. "Side effects of erasing concepts from diffusion models."
Reviewer_2CkaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GIFT, a method designed to suppress target concepts in diffusion models while making them more resistant to relearning through subsequent malicious fine-tuning. GIFT combines gradient ascent on the target-concept denoising loss with representation noising of intermediate attention activations. To preserve safe-concept generation, it employs a denoising-based preservation loss and activation matching with a frozen base model, along with gradient surgery to mitigate conflicts between erasure and preservation gradients. Experiments across 14 artistic styles and 10 ImageNet object classes report stronger resistance to fine-tuning attacks than the evaluated concept erasure and immunization baselines.

Critical strengths

The paper clearly formulates the problem of resisting concept relearning through malicious fine-tuning while preserving safe-concept generation. It evaluates not only initial concept suppression but also resistance to subsequent fine-tuning, explicitly considering the trade-off between erasure and preservation.

The experimental evaluation covers multiple target concepts and includes ablation studies of the proposed components. Tables 1–2 report stronger post-attack resistance than ESD, UCE, and IMMA, while Tables 3–6 examine the effects of individual loss terms, gradient surgery, and trainable parameter choices. Additional experiments evaluate stronger LoRA attacks, DreamBooth, full-parameter fine-tuning, and the SDXL backbone.

Critical weaknesses
  1. The mathematical justification for the representation-noising objective is insufficient. Eq. (5) minimizes the MSE between attention activations and Gaussian noise samples, which is interpreted as Gaussian distribution matching and mutual information reduction. However, if the Gaussian targets are sampled independently of the inputs, the expected MSE reduces to the squared activation norm plus a constant, which is not equivalent to Gaussian distribution matching. Furthermore, the Gaussian–MSE correspondence in Appendix A does not directly establish that this objective reduces representation mutual information or that such a reduction constrains gradients along subsequent fine-tuning trajectories. Additional justification is therefore needed to clarify the relationship between the proposed loss, representation information, and relearning dynamics.

  2. Preservation of semantically related non-target concepts has not been adequately evaluated. It remains unclear whether non-target concepts that are semantically or visually similar to the erased concept are also preserved. For example, after erasing Van Gogh's style or any other target styles simultaneously, can the model still generate images in Monet's style (or other non-target styles)? Although the paper evaluates 14 artistic styles as separate erasure targets, it does not examine whether the resulting models retain their ability to generate other styles. Such cross-concept preservation experiments would help assess the selectivity of GIFT and its potential impact on related non-target concepts.

  3. Evaluation against recent methods and across model architectures remains limited. The main quantitative comparisons focus on ESD, UCE, and IMMA. Although Appendix C.2 includes comparisons with ResAlign and Meta-Unlearning, Figure 6 presents only the cassette-player case, while the concepts covered and the aggregation procedure for the quantitative results in Table 9 are not clearly specified. Clarifying these details would help assess whether GIFT's advantages over recent methods extend beyond the reported setting. The evaluation is also limited to UNet-based models, its applicability to Transformer-based models such as SD3.5 and FLUX.1 remains unclear. This limits the evidence for the architectural generality and practical applicability of the proposed method.

Questions for authors

Q1. Justification for representation noising: If the Gaussian targets in Eq. (5) are sampled independently of the inputs, the expected MSE is equivalent to activation-norm regularization up to a constant. What justifies interpreting this objective as Gaussian distribution matching and mutual information reduction? Is there additional theoretical or empirical evidence that such information reduction limits the reachability of target concepts through subsequent fine-tuning?

Q2. Preservation of related non-target concepts: Does GIFT preserve semantically or visually related concepts after erasing a target concept? Could the authors evaluate cross-concept preservation using the artistic styles or other concepts?

Q3. Generalization across methods and architectures: Could the authors extend the comparisons with ResAlign and Meta-Unlearning in Appendix C.2 to other representative concepts? Could they also evaluate GIFT on a Transformer-based model such as SD3.5 or FLUX.1, or clarify the methodological adaptations required for these architectures?

1 comment
AuthorsOpenReview ↗

# Response to Reviewer 2Cka

We thank the reviewer for their time and constructive feedback.

1. Representation Noising (W1, Q1)

If the Gaussian targets in Eq. (5) are sampled independently of the inputs, the expected MSE is equivalent to activation-norm regularization up to a constant.

We agree. Because η∼N(0,Id)\eta \sim \mathcal{N}(0, I_d) is sampled independently at every step,

Eη ∥z−η∥22=∥z∥22+d,\mathbb{E}_{\eta}\,\lVert z-\eta\rVert_2^2 = \lVert z\rVert_2^2 + d,

where dd is the number of activation coordinates. Eq. (5) therefore penalizes the energy of target-concept activations. It does not match their distribution to a Gaussian, and a small activation norm does not by itself reduce mutual information. We thank the reviewer for highlighting this. We rename the term representation suppression and replace the mutual-information argument in Theorem 1 (Appendix A) with the following.

Consider one attention output projection. Let H⋆H_{\star} be its input features on target-concept inputs, W⋆W_{\star} the defended weights, Z⋆=W⋆H⋆Z_{\star} = W_{\star}H_{\star} the defended outputs, and Z0Z_0 the undefended model's outputs on the same inputs. For any update ΔW\Delta W of rank at most rr, such as a rank-rr LoRA,

\qquad Tail_r(M)=\Big(\sum_{j>r}\sigma_j(M)^2\Big)^{1/2}.$$ *Proof.* $\Delta W H_{\star}$ has rank at most $r$, so the first inequality follows from Eckart–Young–Mirsky. The second holds because $\mathrm{Tail}_r$ is 1-Lipschitz in the Frobenius norm. $\blacksquare$ We minimize $\lVert Z_{\star}\rVert_F$. Once it falls below $\mathrm{Tail}_r(Z_0)$, no rank-$r$ update to this projection can recover the undefended activations. This is a per-layer explanation for rank-limited updates; multi-layer and full fine-tuning remain empirical. ## 2. Preservation of Related Non-Target Concepts(W2, Q2) >Preservation of semantically related non-target concepts has not been adequately evaluated. We ran a cross-concept evaluation for styles and objects. We evaluated the other 13 artists for each of the 14 style (14 * 13 pairs), and the same for objects (10 * 9). Erased-target values are from Tables 1 and 2. We used the same prompts to generate 100 images per concept per pair. | Concept family | Metric | SD1.4 | Non-target, after GIFT | Erased target, after GIFT | | -------------- | --------- | ----- | ---------------------- | ------------------------- | | Objects | ASR-1 | 0.73 | 0.60 | 0.000 | | Objects | ASR-5 | 0.90 | 0.81 | 0.000 | | Objects | CLIP-T | 0.25 | 0.25 | 0.162 | | Objects | Aesthetic | 5.2 | 5.02 | 4.020 | | Styles | CLIP-T | 0.29 | 0.23 | 0.161 | | Styles | Aesthetic | 5.27 | 4.30 | 3.891 | Objects are largely preserved, but styles show substantial collateral damage. One likely cause is the fixed immunization template (“A high quality painting in ⟨artist⟩ style”), whose shared tokens such as “painting” may be affected. We will further discuss this limitation. Collateral effects also arise with concept is filtering from the pre-training data [1]. That is also a known side effect of concept erasure [2]. A surgical isolation is an important future work. ## 3. Baselines and Architectures (W3, Q3) > Evaluation against recent methods and across model architectures remains limited. **Other baselines:** Table 9 evaluates only the cassette-player concept; we will state this in the caption. ResAlign focuses on NSFW content and makes no claims for art styles or objects. Meta-Unlearning is closest to our setting. However, in our cassette-player experiment (Table 9), Meta-Unlearning with its published configuration lost general generation ability before any attack (COCO CLIP-T 0.121). GIFT took 6 minutes per concept, against 1.9 hours for ResAlign and 8.2 hours for Meta-Unlearning on the same GPU. We are happy to extend the comparison. **Architectures:** GIFT targets UNet-based models. Diffusion Transformers such use joint-attention blocks with no separate cross-attention module, so our design (Table 5) has no direct counterpart. They also train with a flow-matching instead of denoising. We do not claim generality beyond the evaluated models and will state this scope explicitly. We hope these results and revisions address the reviewers concerns, and would be glad to clarify further. [1] Creţu, Ana-Maria, et al. "Evaluating concept filtering defenses against child sexual abuse material generation by text-to-image models.". [2] Saha, Shaswati, et al. "Side effects of erasing concepts from diffusion models."

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.