acceptodds
Under review as a conference paper at ICLR 2027

LIFT: Feature Transport for Utility-Preserving CLIP Unlearning

Abstract

As vision-language models such as CLIP are increasingly deployed in real-world applications, model owners may need to remove sensitive or harmful visual concepts without retraining from scratch. Machine unlearning provides a natural framework for this goal: removing targeted information while preserving the model's general utility. However, suppressing a target class is not enough: the target can remain recoverable from visual features with only a few examples, and aggressive unlearning can degrade model performance on ordinary visual recognition tasks, including out-of-distribution (OOD) recognition. We propose LIFT, a Label-Indistinguishable Feature Transport method for utility-preserving CLIP class unlearning. LIFT treats unlearning as controlled feature relocation: instead of pushing target-class features toward arbitrary directions, it transports them toward cluster-specific distributions of real non-forget image features. These anchor distributions provide a natural destination in the original CLIP feature space. Across fine-grained unlearning benchmarks and comparisons with the most recent unlearning methods, LIFT suppresses all target zero-shot accuracy while making the forgotten classes much harder to recover. For instance, LIFT preserves original CLIP's zero-shot transfer ability while achieving a 48.4% relative OOD improvement over the prior state-of-the-art methods. We further evaluate a more challenging selective setting in which only a subset of a dataset's classes is forgotten, while the retained classes are semantically related to the forgotten ones and close to them in CLIP's embedding space. On several datasets spanning car manufacturers, food, remote-sensing scenes, textures, and common objects, LIFT reduces forget accuracy to at most 1.2% while preserving strong accuracy on the retained classes, whereas prior state-of-the-art unlearning methods leave the forgotten classes recognizable or substantially degrade the retained ones.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine unlearning, class unlearning, zero-shot transfer, out-of-distribution generalization, CLIP, vision-language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nz4QOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces LIFT, a utility-preserving CLIP class unlearning method that frames unlearning as controlled feature relocation. Instead of pushing target features toward arbitrary or collapsed spaces, LIFT uses K-means clustering on forget features and matches them via Maximum Mean Discrepancy (MMD) to cluster-specific anchor distributions derived from a generic retain pool. Combined with a retain-inactive LoRA adapter and auxiliary text/retain losses, LIFT aims to suppress target zero-shot accuracy while preserving OOD generalizability and resisting few-shot recovery.

Critical strengths

First, an innovative problem formulation. Shifting unlearning from arbitrary parameter erasure or penalties to distribution-level feature transport is a novel direction. Second, rigorous and comprehensive evaluation. The evaluation goes beyond basic target accuracy, covering OOD zero-shot transfer, few-shot recovery attacks, linear probing, prompt attacks, and complex selective forgetting scenarios.

Critical weaknesses

Regarding the risk of false positives on semantically adjacent classes. During the process of pulling forget features toward generic anchor distributions, is there a risk of boundary erosion for visually or semantically similar non-forget classes? Although the authors evaluate "Near-Retain" accuracy in selective forgetting experiments, it is recommended that the authors further clarify or provide specific analysis on false-positive rate changes to demonstrate whether this feature transport induces increased false alarms on neighboring classes.

Questions for authors

none

1 comment
AuthorsOpenReview ↗

Responses to Reviewer nz4Q

We thank the reviewer for the encouraging comments and the valuable suggestions, which have definitely helped us improve the paper. We have carefully updated the revised manuscript to include all of your suggestions and changes. We further clarify your questions as follows:

W: Regarding the risk of false positives on semantically adjacent classes. During the process of pulling forget features toward generic anchor distributions, is there a risk of boundary erosion for visually or semantically similar non-forget classes? Although the authors evaluate "Near-Retain" accuracy in selective forgetting experiments, it is recommended that the authors further clarify or provide specific analysis on false-positive rate changes to demonstrate whether this feature transport induces increased false alarms on neighboring classes.

Thanks for your suggestion. Following it, we have extended our evaluation to measure false-positive rates (FPR) on neighboring retained classes across 16 selective forgetting requests, including comparisons with additional baselines such as Boundary Shrink, SCAR, and DIET (more detail in Appendix O, Tables 37–38).

Following our Near-Retain evaluation, neighboring classes are defined as the three nearest retained classes of each forgotten class. Each image is classified over the full label set, including forgotten classes. We calculate FPR for each neighboring class and average across classes and requests. We report FPR on retained images only, to assess changes to retained-class boundaries, and on all images, including forgotten ones.

False-positive analysis averaged over 16 selective forgetting requests (%, three-seed means).

MethodForget ↓Near recall ↑Near FPR (retained) ↓Near FPR (all) ↓
Original CLIP71.665.50.770.74
Boundary Shrink0.632.02.853.02
SCAR0.02.72.582.58
DIET5.422.12.983.13
SLUG6.828.01.341.41
CLIPErase1.07.61.231.25
DAMP48.366.20.771.06
LIFT (ours)1.565.80.881.03

The results show that LIFT reduces Forget accuracy from 71.6% to 1.5% while preserving neighboring-class recall (65.8% versus 65.5% for Original CLIP). Importantly, FPR on retained images remains close to Original CLIP (0.88% versus 0.77%), with only minor changes on 15 of the 16 requests. EuroSAT shows the largest increase (7.68% to 9.12%), as detailed in Appendix O (Table 37).

When forgotten images are included, FPR increases from 0.74% to 1.03%, partly because some forgotten images are reassigned to neighboring classes (27.3% versus 10.5% for Original CLIP). However, this increase is much smaller when considering retained images only, indicating that LIFT largely preserves neighboring-class recognition.

Compared with the baselines, LIFT is the only evaluated method achieving low average Forget accuracy while maintaining neighboring-class recall close to Original CLIP. Other methods with strong forgetting preserve only 2.7–32.0% neighboring-class recall, whereas DAMP maintains recall but leaves 48.3% Forget accuracy.

These findings demonstrate that LIFT substantially reduces target recognition while largely preserving neighboring-class boundaries, with only a limited increase in false positives on retained images. We have added this analysis to Section 6 (paragraph "False positives on neighboring classes") and the per-request results to Appendix O.

Reviewer_xF1wOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a new method for class unlearning on CLIP. The proposed method, LIFT, pushes forget-class features toward cluster-specific distributions of non-forget image features. LIFT computes the centroid of forget-class features via k-means and anchors from the retain data, then computes MMD to update the added LoRA adapter to push forget-class features toward the anchors. Experiments show that LIFT makes the forgotten classes much harder to recover while preserving OOD zero-shot behavior.

Critical strengths
  • Comprehensive experiments are conducted on CLIP.
  • The proposed method performs well on OOD.
Critical weaknesses
  • The main idea of pushing target-class features toward cluster-specific distributions of real non-forget image features seems not new; see Boundary unlearning [1], SCAR [2], and DIET [3].
  • The proposed method LIFT needs to build a separate anchor distribution for each forget-feature cluster, which seems impractical (needs to feed forget and retain images to the model every time).
  • As for Section 4, equations (8-9) (an upper bound on the inter-class MMD) seem unable to help explain why these choices reduce recoverable class structure.
  • The paper claims that the proposed method weakens recoverable forget-class geometry, however, if only the added LoRA adapters are trainable, the knowledge about the forgetting class still exists in the visual encoder as well as the text encoder, which could be easily recovered (such as via linear probe from the encoders).

[1] Chen M, Gao W, Liu G, et al. Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary. CVPR 2023.

[2] Bonato J, Cotogni M, Sabetta L. Is retain set all you need in machine unlearning? restoring performance of unlearned models with out-of-distribution images. ECCV 2024. [3] Kunananthaseelan N, Wu J, Le T, et al. DIET: Machine Unlearning on a Data-Diet. AAAI 2026.

Questions for authors
  • In Figure 1 and its caption, the visual encoder remains frozen, and only the adapter is trainable, but in the text (lines 76-77 and lines 147-148) "LIFT updates only a small part of the visual encoder", the LoRA adapters are added into the last two visual blocks. Please clarify this (are only the LoRA adapters trainable, or is a small part of the visual encoder?).
  • Please clarify why equations (8-9) can help explain how these choices reduce recoverable class structure.
  • Please justify the novelty of the main idea compared with [1-3].
  • Would the hyperparameters (such as λ1,λ2,τtext\lambda_1, \lambda_2, \tau_{\texttt{text}} etc.) need be searched again for different models and datasets?
6 comments
AuthorsOpenReview ↗

Responses to Reviewer xF1w [1/3]

We thank the reviewer for the valuable review and comments, which have definitely helped us improve the quality of the paper. We have carefully updated the revised manuscript and uploaded a revision to accommodate all the mentioned suggestions and changes. We further clarify each of your questions as follows:

W1 & Q3: The main idea of pushing target-class features toward cluster-specific distributions of real non-forget image features seems not new; see Boundary unlearning [1], SCAR [2], and DIET [3]. Please justify the novelty of the main idea compared with [1–3].

Thanks for the comment. We clarify that LIFT differs through shared distribution-level feature transport and retain-aware updates, designed to achieve three goals together: strong forgetting, low recoverability, and preservation of CLIP's general utility.

First, Boundary Unlearning [1] uses replacement labels or a temporary shadow class, while DIET [3] assigns forget embeddings to mismatched text-derived prototypes. SCAR [2] moves individual forget images toward the nearest wrong-class feature distribution using Mahalanobis distance.

In contrast, LIFT matches each forget-class feature distribution within a visual cluster to the same distribution of real non-forget CLIP image features using MMD. Anchors are selected by visual proximity, and different clusters use their own nearby destinations rather than a single global destination. Unlike SCAR's per-image matching, LIFT encourages forget classes within the same cluster to become less distinguishable by reducing an upper bound on their inter-class MMD (Eq. 9).

Our controlled ablations support this design. With identical anchors, replacing MMD with pointwise cosine or MSE matching increases four-shot recovery from 4.72–8.20% to 18.59–35.68%, despite similar OOD accuracy (Table 2). Per-image anchor matching also increases average recovery from 7.02% to 14.95% (Table 7) and reduces Near-Retain accuracy on all four tested selective requests (Table 14).

Second, LIFT's retain-inactive adapter helps preserve nearby classes. Its input directions are selected from forget and retain activations and fixed before training, while only its output factor is learned. Replacing it with standard LoRA reduces Near-Retain accuracy from 85.1% to 81.8% on Food-101, 58.8% to 54.6% on RESISC45, and 85.3% to 78.7% on DomainNet sketch (Table 14).

Finally, we compare these methods on same requests as below:

Whole-group forgetting: average over Dogs, CUB, and Cars (%, Table 22).

MethodTarget-ZS ↓GDA@4 ↓OOD-ZS ↑
Original CLIP59.9156.9854.33
Boundary Shrink9.0214.6417.64
Boundary Expanding1.005.584.67
SCAR0.631.061.17
DIET0.623.151.37
LIFT (ours)0.587.0253.88

Selective forgetting: average over 12 matched requests (%, Tables 23–25).

MethodForget ↓Retain ↑Near ↑GDA@4 ↓OOD ↑
Original CLIP65.366.763.978.154.3
Boundary Shrink0.329.330.048.926.7
SCAR0.02.53.318.61.0
DIET4.520.821.564.543.4
LIFT (ours)1.166.963.733.054.0

The results show a clear difference in the recovery–utility trade-off. SCAR and DIET achieve lower whole-group recovery, but their OOD accuracy falls to 1.17% and 1.37%, compared with 53.88% for LIFT. On the 12 selective forgetting requests, SCAR preserves only 2.5% Retain and 1.0% OOD accuracy, while LIFT maintains both close to Original CLIP. DIET also substantially reduces retained-class accuracy (20.8%). Boundary Expanding shows a similar utility trade-off on Audi and Food-101 (Appendix J).

We have added this clarification to Section 2, with more details in Appendix J.

W2: The proposed method LIFT needs to build a separate anchor distribution for each forget-feature cluster, which seems impractical (needs to feed forget and retain images to the model every time).

Thanks for raising this point. We clarify that LIFT constructs the clusters and anchor distributions only once per unlearning request, not at every training step or during inference. We pass the forget and non-forget images through the frozen visual blocks once and cache their features. Clustering, anchor retrieval, and adapter-basis construction then operate on these cached features. LIFT is also computationally efficient, requiring approximately 15 minutes of training for Cars on a single NVIDIA L4 GPU.

LIFT uses a relatively small pool of non-forget images. Cached anchor features can be reused across requests, and Appendix H shows that Oxford Flowers can replace Caltech-101 as the anchor source. Once unlearning is complete, inference requires no anchor images, clustering or retrieval. Anchor construction is confined to the unlearning stage.

We have added this clarification to Section 3.2 and Appendix P.

AuthorsOpenReview ↗

Responses to Reviewer xF1w [2/3]

W3 & Q2: As for Section 4, equations (8–9) (an upper bound on the inter-class MMD) seem unable to help explain why these choices reduce recoverable class structure. Please clarify why equations (8–9) can help explain how these choices reduce recoverable class structure.

Thanks for the question. Eq. (9) explains why shared distribution matching can weaken class-specific feature differences, while Eq. (8) motivates selecting nearby anchors rather than arbitrary destinations. Our ablations further provide direct experimental evidence supporting these choices.

Let Pc,sP_{c,s} and Pc′,sP_{c',s} denote the transformed feature distributions of two forget classes within cluster ss, and let QsQ_s be their shared non-forget anchor distribution. Eq. (9) gives

MMD⁡(Pc,s,Pc′,s)≤MMD⁡(Pc,s,Qs)+MMD⁡(Pc′,s,Qs).\operatorname{MMD}(P_{c,s},P_{c',s}) \leq \operatorname{MMD}(P_{c,s},Q_s) + \operatorname{MMD}(P_{c',s},Q_s).

First, MMD measures the largest difference in expected scores over unit-norm functions in the kernel's RKHS [1]. By reducing each class's discrepancy from the same anchor distribution, LIFT tightens an upper bound on the differences between forget classes within a cluster.

Moreover, our mixture of Gaussian RBF kernels is characteristic [2]. Under exact population matching, different forget classes within a cluster have identical final-feature distributions and cannot be distinguished using those features within that cluster. In practice, matching is approximate, and information may remain across clusters. Thus, Eq. (9) provides a theoretical basis for reducing recoverable class differences through shared distribution matching, without implying complete erasure.

Importantly, our ablations provide direct evidence for this mechanism. With the same cluster-selected anchors, replacing MMD with pointwise cosine or MSE matching increases four-shot recovery from 4.72–8.20% to 18.59–35.68%, despite similar OOD accuracy (Table 2). This shows that providing non-forget destinations alone is insufficient. How the distributions are matched matters. Table 8 further shows that class transport alone achieves the lowest four-shot recovery among the evaluated loss combinations across Dogs, CUB, and Cars.

T = class transport, R = retain distillation, F = forget-text regularizer.

ObjectiveTarget-ZS ↓GDA@4 ↓OOD-ZS ↑
R only57.1553.9054.32
T only0.865.1753.58
T + R1.075.8053.93
T + R + F (LIFT)0.587.0253.88

With the architecture and anchor construction unchanged, class transport alone reduces recovery to 5.17%, compared with 53.90% for retain distillation alone, while maintaining 53.58% OOD accuracy. This confirms that the class-transport objective, rather than retain distillation, is the principal source of recovery reduction in this ablation.

Eq. (8) addresses the other important consideration: preserving the original representation. Its bound relates the distance between an original forget feature and an anchor to the within-cluster spread and anchor-to-centroid distance. K-means reduces the former on average, while nearest-anchor retrieval reduces the latter. This motivates local destinations, although the bound does not itself guarantee retained-class preservation.

Our selective ablations show why additional protection is necessary. On Audi and Food-101, class transport alone achieves 13.5% and 20.4% recovery, but Near-Retain accuracy decreases to 34.7% and 57.8%. Adding retain distillation restores Near-Retain to 74.0% and 85.9%, nearly identical to Original CLIP (74.5% and 85.9%), while keeping recovery substantially reduced (Appendix M, Table 33).

These results highlight an important challenge in CLIP unlearning: reducing recoverable class information is insufficient if nearby retained classes or general zero-shot recognition are also damaged. LIFT addresses both requirements through shared distribution-level transport and retain-aware preservation.

We have added this explanation to Section 4 in the revised manuscript.

[1] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A Kernel Two-Sample Test. JMLR 2012.

[2] Bharath K. Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Bernhard Schölkopf, and Gert R. G. Lanckriet. Hilbert Space Embeddings and Metrics on Probability Measures. JMLR 2010.

AuthorsOpenReview ↗

Responses to Reviewer xF1w [3/3]

W4: The paper claims that the proposed method weakens recoverable forget-class geometry, however, if only the added LoRA adapters are trainable, the knowledge about the forgetting class still exists in the visual encoder as well as the text encoder, which could be easily recovered (such as via linear probe from the encoders).

Thank you for the comment. We clarify that LIFT trains only the LoRA adapter output factors in the last two visual blocks and the final visual LayerNorm. The trained adapters and LayerNorm remain active during inference and linear probing. Our stronger linear-probe experiments demonstrate that LIFT substantially reduces forget-class recoverability from its adapted visual representation while preserving CLIP's general zero-shot utility.

Following your suggestion, we tested stronger layer-wise probes as below (More details in Appendix N):

(Linear-probe recovery accuracy (%) on forgotten classes. Block 9 is the last unchanged visual block. OOD-ZS measures the utility of the unlearned model before probing. The complete table also includes block 10 and two selective requests.)

MethodBlock 9 DogsBlock 9 CUBBlock 9 CarsOutput DogsOutput CUBOutput CarsOOD-ZS ↑ (Dogs / CUB / Cars)
Original CLIP46.342.040.976.277.485.854.33
SLUG46.342.040.975.577.685.75.02 / 22.04 / 37.44
CLIPErase48.430.842.553.736.253.92.33 / 7.06 / 7.86
DAMP44.740.435.761.162.968.542.52 / 31.35 / 35.08
DIET20.613.814.319.312.812.81.27 / 1.38 / 1.46
SCAR2.92.81.92.92.71.81.17 / 1.45 / 0.88
Boundary Shrink28.918.113.437.929.524.822.83 / 15.32 / 14.77
LIFT (ours)46.342.040.944.534.435.254.09 / 53.49 / 54.06

At the output, recovery falls from 76.2/77.4/85.8% to 44.5/34.4/35.2% on Dogs/CUB/Cars, while LIFT's OOD accuracy remains close to Original CLIP. These probes recover more than those in the original Table 9. DIET, SCAR, and Boundary Shrink recover less, but their OOD accuracy is only 0.88–22.83%. Block-9 recovery is identical to Original CLIP, as expected for unchanged layers. LIFT's final output is less recoverable than block 9 on the three whole-group requests, demonstrating that the adapted visual features reduce recoverable class information beyond merely suppressing text-classifier predictions. Earlier visual layers remain unchanged by design to preserve pretrained knowledge.

Under fully supervised probing, LIFT achieves 95.6%/96.7% recovery on Food-101/DomainNet, below Original CLIP (98.3%/99.1%), SLUG (98.2%/98.8%), and DAMP (98.2%/98.5%). CLIPErase achieves lower recovery (9.1%/91.8%), but its OOD accuracy drops to 0.8%/4.1%, compared with 54.5%/53.4% for LIFT (Tables 20 and 12). This further demonstrates the importance of reducing recoverability while preserving CLIP's general utility.

We have added this evaluation to Appendix N (Table 36), with a short summary in Section 5.2.

Q1: In Figure 1 and its caption, the visual encoder remains frozen, and only the adapter is trainable, but in the text (lines 76-77 and lines 147-148) "LIFT updates only a small part of the visual encoder", the LoRA adapters are added into the last two visual blocks. Please clarify this (are only the LoRA adapters trainable, or is a small part of the visual encoder?).

Thanks for pointing this out. LIFT inserts trainable LoRA adapters into the last two visual blocks while keeping all original visual-block weights frozen. Only the adapter output factors and the final visual LayerNorm are trained, approximately 223K trainable parameters in total for CLIP ViT-B/16. All pretrained visual-block weights, the adapter input bases, the visual-text projection, and the entire text encoder remain frozen. We have added this clarification to the Figure 1 caption, Sections 3.1 and 3.3, and Appendix P.

Q4: Would the hyperparameters (such as λ1\lambda_1, λ2\lambda_2, τtext\tau_{text} etc.) need be searched again for different models and datasets?

Thanks for the question. We use the same hyperparameters across all evaluated datasets, forgetting requests, and both CLIP ViT-B/16 and ViT-L/14, without dataset- or backbone-specific tuning. Specifically, we use λ1 = 1, λ2 = 3, τ_text = 0.05, K = 16, M = 256, rank = 16, and train for 800 steps with a learning rate of 2×10⁻⁴.

We have also provided extensive hyperparameter sensitivity analyses in Tables 10, 13, 15, and 34–35. Across the tested settings, Retain and Near-Retain accuracy remain stable, while target suppression remains strong. Although recovery varies more with the number of clusters and anchors, LIFT maintains the same default configuration across all experiments rather than tuning it for individual requests. We have added this clarification to Section 5.1 and Appendix P.

Reviewer_o8wsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LIFT, a feature transport method for CLIP class unlearning that aims to reduce forget-class information while preserving model utility. The authors evaluate the method across several datasets and two CLIP backbones, considering zero-shot forgetting, few-shot recovery, OOD generalization, and sequential unlearning. The results show promising performance in forgetting target classes while maintaining retain and OOD accuracy.

Critical strengths
  1. The proposed feature transport approach is interesting, as it aims to reduce the recoverability of forget-class information while preserving the model's utility. This offers a different perspective on CLIP class unlearning beyond simply suppressing forget-class predictions.

  2. The experimental evaluation is relatively comprehensive, covering several datasets and two CLIP ViT backbones. The authors consider zero-shot forgetting, few-shot recovery, OOD generalization, selective forgetting, and sequential unlearning. The results show a promising balance between forgetting effectiveness and preserving model utility.

Critical weaknesses

1. Incomplete related work. The paper does not sufficiently discuss existing CLIP class-unlearning methods, and some relevant studies on relearning and recovery are not cited. Including these works would help clarify the research gap and better motivate the proposed method.

2. Missing relevant baselines and dataset comparisons. Although the authors compare LIFT with SLUG, CLIPErase, and DAMP, several existing CLIP-specific class-unlearning methods are not included. Moreover, Caltech-101 and Oxford Flowers are not considered as class-forgetting benchmarks. Including additional baselines and evaluating LIFT on these datasets would allow for more direct comparisons with previous CLIP class-unlearning methods and provide stronger evidence for the advantages of LIFT.

Questions for authors
  1. Could the authors explain how LIFT differs from other recent CLIP class-unlearning methods that are not discussed in detail, and whether these methods could provide relevant additional baselines?

  2. Although the paper evaluates several recovery attacks, how robust is LIFT against stronger adaptive relearning attacks or larger recovery budgets? Would LIFT still outperform the existing methods under these more challenging conditions?

  3. While the paper reports retain accuracy in the selective forgetting experiments, some ablation and sensitivity studies (Tables 8, 10, and 13) only report forgetting, recovery, and OOD accuracy. Could the authors also conduct these ablations under selective forgetting and report retain accuracy? Since LIFT aims to preserve retained knowledge while forgetting target classes, it would be useful to understand how the different loss terms and hyperparameter choices affect the trade-off between forget and retain accuracy.

8 comments
AuthorsOpenReview ↗

Responses to Reviewer o8ws [1/6]

We thank the reviewer for the detailed and constructive comments, which have definitely helped us improve the quality of the paper. We have carefully revised our manuscript and uploaded a revision to accommodate all your mentioned suggestions. Below, we further clarify each of your questions in detail:

W1 & Q1: Incomplete related work. The paper does not sufficiently discuss existing CLIP class-unlearning methods, and some relevant studies on relearning and recovery are not cited. Including these works would help clarify the research gap and better motivate the proposed method. Could the authors explain how LIFT differs from other recent CLIP class-unlearning methods that are not discussed in detail, and whether these methods could provide relevant additional baselines?

Thanks for the suggestion. Following it, we have expanded Section 2 to cover Lip [1], TIS [2], DIET [3], SCAR [4], Boundary Unlearning [5], and relearning/recovery studies. Appendix J details their differences from LIFT. W2 and Q2 provide new baseline and recovery experiments. Below, we clarify LIFT's key distinctions from these methods.

We clarify that LIFT differs in three aspects: shared distribution-level transport, retain-aware updates, and a holistic evaluation of forgetting, recoverability, and general CLIP utility.

First, LIFT matches each forget-class feature distribution within a visual cluster to the same distribution of real non-forget CLIP image features using MMD. Anchors are selected by visual proximity rather than a retained-class label, and different clusters use different nearby destinations. Sharing the same destination encourages forget classes within each cluster to become less distinguishable, as motivated by the inter-class MMD bound (Eq. 9).

TIS changes the text projection [2], while Lip regularizes both encoders using synthetic samples [1]. Boundary Unlearning uses replacement labels or a shadow class [5]. SCAR matches individual forget images toward a wrong-class feature distribution using Mahalanobis distance [4], while DIET assigns embeddings to mismatched text-derived prototypes through optimal transport [3]. LIFT's distinction is shared class-cluster distribution-to-distribution matching, not simply choosing a distribution as a destination.

Our controlled ablations show why distribution-level matching matters, beyond simply choosing non-forget anchors as destinations. With identical cluster-selected anchors, replacing MMD with pointwise cosine or MSE matching raises four-shot recovery from 4.72–8.20% to 18.59–35.68% at similar OOD accuracy (Table 2). Replacing class-cluster matching with per-image anchor matching raises average recovery from 7.02% to 14.95% (Table 7) and lowers Near-Retain accuracy on all four tested selective requests (Table 14).

Second, LIFT uses a retain-inactive adapter, whose input directions are selected from forget and retain activations and fixed before training, while only its output factor is learned. This favors responses to forget images over retained ones, complementing retain distillation. Under the same training objective, replacing it with standard LoRA reduces Near-Retain accuracy from 85.1% to 81.8% on Food-101, 58.8% to 54.6% on RESISC45, and 85.3% to 78.7% on DomainNet sketch (Table 14).

Third, LIFT evaluates recoverability alongside nearby-class retention and held-out OOD utility. This distinction matters because low recovery can also result from general damage to CLIP's pretrained representation. Across 12 selective requests, LIFT reduces Forget accuracy from 65.3% to 1.1% and four-shot recovery from 78.1% to 33.0%. Meanwhile, Retain, Near-Retain, and OOD remain 66.9% / 63.7% / 54.0%, versus 66.7% / 63.9% / 54.3% for Original CLIP (W2). The additional baseline comparisons in W2 further show why forgetting, recovery, and general utility must be evaluated together.

We have incorporated these additional baseline comparisons and supporting analyses into the revised manuscript.


References:

[1] Alexey Kravets and Vinay P. Namboodiri. Zero-Shot Class Unlearning in CLIP with Synthetic Samples. WACV 2025.

[2] Alexey Kravets and Vinay P. Namboodiri. Zero-shot CLIP Class Forgetting via Text-image Space Adaptation. TMLR 2025.

[3] Kunananthaseelan, Nilakshan, et al. DIET: Machine Unlearning on a Data-Diet. AAAI 2026.

[4] Jacopo Bonato, Marco Cotogni, and Luigi Sabetta. Is Retain Set All You Need in Machine Unlearning? Restoring Performance of Unlearned Models with Out-of-Distribution Images. ECCV 2024.

[5] Chen, Min, et al. Boundary Unlearning: Rapid Forgetting of Deep Networks via Shifting the Decision Boundary. CVPR 2023.

AuthorsOpenReview ↗

Responses to Reviewer o8ws [2/6]

W2: Missing relevant baselines and dataset comparisons. Although the authors compare LIFT with SLUG, CLIPErase, and DAMP, several existing CLIP-specific class-unlearning methods are not included. Moreover, Caltech-101 and Oxford Flowers are not considered as class-forgetting benchmarks. Including additional baselines and evaluating LIFT on these datasets would allow for more direct comparisons with previous CLIP class-unlearning methods and provide stronger evidence for the advantages of LIFT.

Following your suggestion, we have additionally evaluated the following baselines: TIS, Lip, DIET, SCAR, and Boundary Unlearning (Boundary Shrink and Boundary Expanding), using identical requests, splits, and evaluation settings. Results are three-seed means.

a) Whole-group forgetting

Three-seed means. Table 22 in Appendix J.

MethodTarget-ZS ↓ (Dogs / CUB / Cars)OOD-ZS ↑ (Dogs / CUB / Cars)GDA@4 ↓ (Dogs / CUB / Cars)
Original CLIP59.12 / 55.09 / 65.5154.33 / 54.33 / 54.3344.00 / 62.43 / 64.50
SLUG (Table 1)1.60 / 3.63 / 11.045.02 / 22.04 / 37.4427.09 / 58.94 / 60.66
CLIPErase (Table 1)0.76 / 0.58 / 0.632.33 / 7.06 / 7.8612.19 / 11.34 / 11.74
DAMP (Table 1)4.36 / 6.13 / 7.0842.52 / 31.35 / 35.0816.51 / 34.18 / 28.20
TIS (new)25.24 / 36.76 / 52.6451.80 / 53.48 / 53.9144.00 / 62.43 / 64.50
DIET (new)0.77 / 0.55 / 0.531.27 / 1.38 / 1.463.54 / 3.18 / 2.74
SCAR (new)0.79 / 0.52 / 0.581.17 / 1.45 / 0.881.34 / 1.13 / 0.72
Boundary Shrink (new)8.99 / 6.18 / 11.9022.83 / 15.32 / 14.7714.51 / 14.91 / 14.50
Boundary Expanding (new)1.32 / 1.20 / 0.471.75 / 11.40 / 0.861.62 / 14.51 / 0.62
LIFT (ours)0.76 / 0.53 / 0.4654.09 / 53.49 / 54.068.15 / 8.20 / 4.72

LIFT is the only method shown that keeps target accuracy below 1% on all three datasets while preserving OOD accuracy within 1% in absolute accuracy of Original CLIP. TIS preserves OOD accuracy but leaves substantial target recognition and unchanged visual recovery. DIET, SCAR, and Boundary Expanding can achieve lower recovery, accompanied by severe utility loss. LIFT combines target suppression with four-shot recovery of 4.72–8.20% and OOD accuracy of 53.49–54.09%.

b) Selective forgetting

Boundary Shrink, SCAR, and DIET are evaluated on all 12 requests across seven datasets. TIS on six car-manufacturer requests. Lip and Boundary Expanding on Audi and Food-101. Each group compares methods over the same requests. Complete results are in Appendix J, Tables 23–25.

Method (evaluated requests)Forget ↓Retain ↑Near ↑GDA@4 ↓OOD ↑COCO R@1 ↑
Original CLIP (12 requests)65.366.763.978.154.333.4
Boundary Shrink (12 requests)0.329.330.048.926.717.7
SCAR (12 requests)0.02.53.318.61.00.0
DIET (12 requests)4.520.821.564.543.430.5
LIFT (12 requests)1.166.963.733.054.032.0
Original CLIP (6 car manufacturers)56.166.262.764.054.333.4
TIS (6 car manufacturers)26.959.552.964.053.333.3
LIFT (6 car manufacturers)0.365.861.717.154.233.0
Original CLIP (Audi and Food-101)67.976.380.271.854.333.4
Lip (Audi and Food-101)52.161.964.170.846.632.1
Boundary Expanding (Audi and Food-101)0.11.51.014.42.61.4
LIFT (Audi and Food-101)0.776.179.122.654.332.3

Across the 12 requests, LIFT preserves Retain, Near-Retain, and OOD accuracy near Original CLIP while reducing target recognition and recovery. SCAR achieves lower recovery but retains only 2.5% Retain and 1.0% OOD accuracy. Boundary Expanding shows the same trade-off on its two requests. DIET has higher COCO retrieval on 8 requests, but Retain accuracy on those requests is only 5.0–32.1%. These comparisons support the significance of evaluating recovery and utility together.

AuthorsOpenReview ↗

Responses to Reviewer o8ws [3/6]

c) Caltech-101 and Oxford Flowers

For each dataset, we evaluate two ten-class requests, selecting nearest-neighbor or random classes, over three seeds and with the same LIFT configuration. Caltech-101 forgetting uses Oxford Flowers as the anchor pool and Aircraft/UCF-101 for OOD evaluation. Dogs is excluded since Caltech-101 includes a Dalmatian category which may overlap with Dogs classes. Flowers forgetting additionally uses Dogs for OOD evaluation. All methods share the OOD suite within each request (Appendix K, Tables 26–27).

Caltech-101 (OOD: FGVC Aircraft and UCF-101)

MethodForget ↓Retain ↑Near ↑GDA@4 ↓OOD ↑COCO R@1 ↑
Original CLIP91.091.382.998.846.133.4
SLUG7.823.09.298.634.815.3
CLIPErase3.647.725.189.517.018.9
DAMP53.191.583.697.745.633.4
TIS16.889.580.298.844.833.6
DIET12.332.629.194.426.926.5
SCAR0.01.12.417.20.90.0
Boundary Shrink1.643.041.456.120.414.6
LIFT (ours)5.791.583.551.445.929.8

Oxford Flowers (OOD: FGVC Aircraft, UCF-101, and Stanford Dogs)

MethodForget ↓Retain ↑Near ↑GDA@4 ↓OOD ↑COCO R@1 ↑
Original CLIP90.067.357.597.950.533.4
SLUG5.217.715.197.739.814.0
CLIPErase0.024.313.487.630.424.1
DAMP78.768.058.896.450.133.4
TIS0.160.149.997.947.533.4
DIET3.620.918.791.447.834.0
SCAR0.01.20.028.00.90.0
Boundary Shrink1.634.834.571.624.316.7
LIFT (ours)0.068.260.634.650.332.6

LIFT reduces mean Forget accuracy to 5.7% on Caltech-101 and 0.0% on Flowers, while retaining recognition and OOD accuracy near Original CLIP. Recovery falls from 98.8% to 51.4% and from 97.9% to 34.6%, respectively. The hardest Caltech request retains 11.3% Forget accuracy. TIS and Boundary Shrink suppress it further but remain more recoverable (97.9% and 93.3%, versus 58.1% for LIFT). The new benchmarks therefore support LIFT's forgetting–utility balance without dataset-specific tuning.

We have added these new baselines to Section 5, Section 6 and Appendix J (Tables 22–25). The Caltech-101 and Oxford Flowers results are in Appendix K (Tables 26–27)

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all three reviewers for their careful assessments, constructive comments, and valuable suggestions. Their feedback has helped us strengthen the experimental evaluation, clarify the methodological contributions, and improve the overall quality of our paper. We have carefully revised the manuscript, with the new and modified content highlighted in blue. The main additions and clarifications are summarized below.

  1. Expanded related work and baseline comparisons. We expanded the discussion of CLIP class-unlearning and recovery methods and added comparisons with five additional methods, including TIS, Lip, DIET, SCAR, and Boundary Unlearning. The new experiments covered both whole-group and selective forgetting under matched evaluation settings, providing further evidence for LIFT's balance between target suppression, reduced recoverability, and preserved CLIP utility (Section 2 and Appendix J, Tables 22–25).

  2. Additional forgetting benchmarks. We extended the evaluation to Caltech-101 and Oxford Flowers, considering both nearest-neighbor and random forgetting requests over three seeds. LIFT reduced average Forget accuracy to 5.7% on Caltech-101 and 0.0% on Oxford Flowers, while preserving Retain, Near-Retain, and OOD accuracy close to Original CLIP, without dataset-specific hyperparameter tuning (Appendix K, Tables 26 and 27).

  3. Stronger recovery attacks and internal representations. We extended recovery evaluations to LoRA relearning with 4, 16, and all training images per forgotten class for up to 2,000 steps, full visual-encoder fine-tuning, retained-only relearning, and stronger layer-wise probes. With 16 shots, LIFT required more steps than SLUG and DAMP to reach 50% recovery on all five requests. Although methods such as DIET and CLIPErase had lower recovery, this came at a substantial cost to damaged retained-class accuracy or general utility. These experiments further demonstrated the importance of evaluating recovery resistance together with preserved CLIP utility (Section 5.2, Appendices L and N, Tables 28–32 and 36).

  4. Extended ablation and sensitivity studies. We conducted additional selective-forgetting ablations on Audi and Food-101 to examine the contributions of class transport, retain distillation, and forget-text regularization. We also evaluated sensitivity to the number of clusters, anchor count, and the forget-text weight, reporting Retain and Near-Retain accuracy alongside forgetting and recovery. The results supported the role of retain-aware updates in preserving nearby classes and the use of a shared hyperparameter configuration across all requests and experiments in our work (Section 5.3 and Appendix M, Tables 33–35).

  5. False positives on neighboring retained classes. We added a false-positive analysis across 16 selective forgetting requests, including comparisons with additional baselines. LIFT preserved neighboring-class recall at 65.8%, compared with 65.5% for Original CLIP, while the false-positive rate on retained images changed from 0.77% to 0.88%. These results provided further evidence of neighboring-class preservation under strong target forgetting (Section 6 and Appendix O, Tables 37 and 38).

  6. Theoretical motivation and implementation clarifications. We clarified LIFT's distinction from previous methods through shared distribution-level matching and retain-aware updates (Section 2 and Appendix J, Table 21). We further explained the motivation behind Eqs. (8–9) (Section 4), clarified the trainable parameters (Sections 3.1 and 3.3, Appendix P), and provided details on anchor construction, feature caching, and anchor-free inference (Section 3.2 and Appendix P).

We hope these additional experiments, analyses, and clarifications address the reviewers' concerns and further strengthen the quality and completeness of our work. We greatly appreciate the reviewers' time and thoughtful feedback. Detailed responses to each reviewer's comments are provided below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.