LIFT: Feature Transport for Utility-Preserving CLIP Unlearning

As vision-language models such as CLIP are increasingly deployed in real-world applications, model owners may need to remove sensitive or harmful visual concepts without retraining from scratch. Machine unlearning provides a natural framework for this goal: removing targeted information while preserving the model's general utility. However, suppressing a target class is not enough: the target can remain recoverable from visual features with only a few examples, and aggressive unlearning can degrade model performance on ordinary visual recognition tasks, including out-of-distribution (OOD) recognition. We propose LIFT, a Label-Indistinguishable Feature Transport method for utility-preserving CLIP class unlearning. LIFT treats unlearning as controlled feature relocation: instead of pushing target-class features toward arbitrary directions, it transports them toward cluster-specific distributions of real non-forget image features. These anchor distributions provide a natural destination in the original CLIP feature space. Across fine-grained unlearning benchmarks and comparisons with the most recent unlearning methods, LIFT suppresses all target zero-shot accuracy while making the forgotten classes much harder to recover. For instance, LIFT preserves original CLIP's zero-shot transfer ability while achieving a 48.4% relative OOD improvement over the prior state-of-the-art methods. We further evaluate a more challenging selective setting in which only a subset of a dataset's classes is forgotten, while the retained classes are semantically related to the forgotten ones and close to them in CLIP's embedding space. On several datasets spanning car manufacturers, food, remote-sensing scenes, textures, and common objects, LIFT reduces forget accuracy to at most 1.2% while preserving strong accuracy on the retained classes, whereas prior state-of-the-art unlearning methods leave the forgotten classes recognizable or substantially degrade the retained ones.