- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GIFT combines gradient ascent on target denoising loss with representation noising to resist malicious fine-tuning. Training on safe data, reference activation matching, and asymmetric gradient projection preserve normal generation. Evaluations cover 14 styles and 10 object classes, SDXL, and multiple attack settings, showing improved resistance under the tested conditions with largely preserved SD1.4 generation quality.
Critical strengths
- The paper addresses a practically issue: erased concepts can be recovered via fine-tuning after weight release.
- The post-attack object ASR-1 drops from 0.450 for IMMA to 0.144 for GIFT, while performance on safe prompts remains close to that of the original SD1.4 model.
Critical weaknesses
- The method combines several established ideas. Although component ablations are provided, a controlled comparison against a simpler activation-suppression baseline is missing, leaving the added value of the full framework unclear.
- The representation-removal mechanism is mainly supported by output metrics and ablations. While object ASR is classifier-based, style metrics remain sensitive to image quality. Representation probing and style-recognition tests would further strengthen this claim.
- Main comparisons still focus on earlier methods, while recent baselines receive only limited discussion in the appendix. A broader multi-concept evaluation of compatible recent methods under a common fine-tuning attack protocol would be more informative.
- Stronger attacks mainly target Van Gogh, with some evaluated only qualitatively. Since both defender and attacker data are synthetic and from the same source, and the metrics are largely aggregate utility measures, practical generalization remains insufficiently tested.
Questions for authors
Although the paper addresses a meaningful problem and reports promising improvements, the added value of the proposed framework and the efficacy of its representation-removal mechanism remain insufficiently validated. In addition, the limited comparison with recent methods and the narrow coverage of stronger attacks weaken the broader claims.
We thank the reviewer for the constructive review
1. Contribution & baseline (W1)
(Novelty). Tamper-resistance for diffusion models is a relatively underexplored problem. Concept-erasure methods are undone by a few hundred fine-tuning steps, and existing immunization methods either still relearn the target under attack (IMMA, ResAlign) or lose general generation ability (Meta-Unlearning). Our main contribution is finding a recipe that works: which signals to apply (loss ascent, activation suppression), where to apply them (both self- and cross-attention, Table 5), and how to keep the rest of the model intact (safe activation anchoring and asymmetric gradient surgery, Table 4). No single component achieves this on its own, but together they resist the attack and preserve safe generation, at 6 minutes per concept.
(Simpler baseline). Because η is sampled independently at every step, Eη∥z−η∥22=∥z∥22+d, so Eq. (5) is equivalent to activation suppression in expectation. We rename the term representation suppression and revise its interpretation accordingly.
The simpler baseline is therefore the "Lnoise only" row of Table 3. Suppression alone lets the attack restore the concept. Ascent alone leaves the concept reachable and lowers safe aesthetic score.
2. Target Erasure and Preservation (W2)
Target outputs: We do not claim to replace the target concept with another concept. GIFT is designed to suppress target generation by producing corrupted outputs, which will be captured by CLIP and AS scores. We explicitly show these outputs in Figs. 1–2. General safe generation is evaluated separately using MS-COCO metrics and qualitative examples.
3. Recent Baselines (W3)
Appendix C.2 compares GIFT with ResAlign and Meta-Unlearning under the same attack protocol. ResAlign focuses on NSFW content and makes no claims for art styles or objects. Meta-Unlearning is closest to our setting. However, in our cassette-player experiment (Table 9), Meta-Unlearning with its published configuration lost general generation ability before any attack (COCO CLIP-T 0.121). GIFT took 6 minutes per concept, against 1.9 hours for ResAlign and 8.2 hours for Meta-Unlearning on the same GPU. We are happy to extend the comparison.
4. Stronger Attacks and Attack Data (W4)
The default 5k-step LoRA attack covers all 14 styles and 10 objects; the hyperparameter sweeps, DreamBooth and full fine-tuning are evaluated on Van Gogh only, which we state explicitly. Immunization and attack sets are disjoint, and the evaluation prompts differ from the immunization template, but both sets are generated by SD1.4. We agree that attacks with real images would better test generalization.
We hope these results and revisions address the reviewer's concerns, and would be glad to clarify further.