AutoAnchor: Stable Diffusion Unlearning Using Cross-Attention as a Manifold Surrogate
Desk rejected at ICLR 2027, so left out of the paper lists and the map.
Abstract
Diffusion unlearning is essential for mitigating the generation of harmful or copyrighted content in text-to-image models. Current diffusion unlearning techniques determine the model update direction by either using alternatives of the target concept as anchors or using empty prompts. The anchor-based method relies on manually and semantically chosen anchors that risk biased unlearning, while the anchor-free method inherently suffers from imprecise unlearning due to unconstrained latent updates. In this work, we theoretically formalize diffusion unlearning under the manifold hypothesis and prove that lacking a manifold-proximal anchor can induce substantial normal-space drift that degrades unlearning performance. Motivated by this analysis, we propose AutoAnchor, a two-stage framework that automatically synthesizes manifold-proximal anchors. However, direct geometric manifold optimization is computationally intractable. To address this challenge, AutoAnchor introduces a novel cross-attention consistency loss, which serves as a highly efficient surrogate for manifold proximity. Experimental results demonstrate that AutoAnchor effectively achieves precise and unbiased unlearning across various state-of-the-art baselines, significantly improving targeted concept removal (by up to 28.17% in CLIP score) and non-target utility (by up to 4.17% in CLIP score). Moreover, AutoAnchor can also be easily integrated into existing diffusion unlearning methods to enhance their unlearning performance (by 34.48% for concept removal and 32.63% for utility on average).
- Keywords:
- diffusion unlearning, concept unlearning, concept erasure