Erase-while-Sparsify: Joint Sparse Training and Concept Erasure for Diffusion Models

Text-to-image diffusion models are often deployed under both prescribed parameter budgets and content-policy constraints requiring the suppression of undesirable or restricted concepts. A common approach is sequential: prune and recover the model before erasure, or erase the target concept before sparsification and recovery. Both orders optimize the stages separately. In the former, erasure is performed on a sparse connectivity pattern selected without its objective; in the latter, subsequent pruning weakens the target–retained separation established by erasure, and recovery without the erasure objective partially restores the target concept. We propose Erase-while-Sparsify (EwS), a unified framework that jointly optimizes sparsification and concept erasure. Rather than fixing the sparse topology or imposing the target sparsity at the outset, EwS progressively reduces model density under joint erasure and preservation objectives, so that the weights adapt to both objectives as the sparse budget is imposed. It applies gradual magnitude pruning and gradient-based regrowth to convolutional layers, while using a concept-ratio criterion to prioritize cross-attention key and value connections with high target-relative weight scores for removal. We evaluate EwS on artistic-style, object-class, and explicit-content erasure. Under matched parameter budgets, we compare EwS with existing U-Net erasure methods applied to a common sparse model as sequential baselines. EwS achieves a favorable overall erasure–preservation trade-off relative to these baselines without requiring separate pre-pruning and recovery stages. Across the evaluated target-density settings from 0.9 to 0.5, EwS retains its advantage over the sequential baselines.