When Compatible Gradients Conflict over Time: Trajectory-Aware Diffusion Model Unlearning

Concept erasure in text-to-image diffusion models must prevent adaptive attacks from recovering an erased concept while preserving neighboring semantics and generation quality. Existing methods largely address this robustness–retention trade-off through loss balancing or single-step gradient surgery, implicitly treating erase–retain interference as a local optimization phenomenon. In this paper, we show that this view is incomplete. Across unsafe-content, object, and artistic-style erasure, erase and retain gradients are nearly orthogonal or weakly aligned at individual steps, yet their accumulated directions become increasingly antagonistic over training. We attribute this reversal to two forms of optimizer-history dependence. Specifically, shared first- and second-moment optimizer states mix unmatched erase and retain gradients across optimization steps, creating trajectory conflict despite same-step compatibility. Meanwhile, fixed optimizer memory cannot accommodate directional changes within each objective trajectory. Based on this diagnosis, we propose a trajectory-aware Robust-Retain Concept Erasure framework (RRCE) that alternates recovery discovery with erase–retain defense. The recovery phase uses textual inversion to discover target representations that can still recover the target concept from the current model. The defense phase suppresses these representations through anchor-guided erasure and preserves non-target behavior through teacher-matching retention. To address the two trajectory-level failures, RRCE maintains objective-specific optimizer states and adapts momentum according to adjacent-gradient similarity within each trajectory. Experiments demonstrate state-of-the-art robustness–retention performance against adaptive recovery attacks.