- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces LIFT, a utility-preserving CLIP class unlearning method that frames unlearning as controlled feature relocation. Instead of pushing target features toward arbitrary or collapsed spaces, LIFT uses K-means clustering on forget features and matches them via Maximum Mean Discrepancy (MMD) to cluster-specific anchor distributions derived from a generic retain pool. Combined with a retain-inactive LoRA adapter and auxiliary text/retain losses, LIFT aims to suppress target zero-shot accuracy while preserving OOD generalizability and resisting few-shot recovery.
Critical strengths
First, an innovative problem formulation. Shifting unlearning from arbitrary parameter erasure or penalties to distribution-level feature transport is a novel direction. Second, rigorous and comprehensive evaluation. The evaluation goes beyond basic target accuracy, covering OOD zero-shot transfer, few-shot recovery attacks, linear probing, prompt attacks, and complex selective forgetting scenarios.
Critical weaknesses
Regarding the risk of false positives on semantically adjacent classes. During the process of pulling forget features toward generic anchor distributions, is there a risk of boundary erosion for visually or semantically similar non-forget classes? Although the authors evaluate "Near-Retain" accuracy in selective forgetting experiments, it is recommended that the authors further clarify or provide specific analysis on false-positive rate changes to demonstrate whether this feature transport induces increased false alarms on neighboring classes.
Questions for authors
none
Responses to Reviewer nz4Q
We thank the reviewer for the encouraging comments and the valuable suggestions, which have definitely helped us improve the paper. We have carefully updated the revised manuscript to include all of your suggestions and changes. We further clarify your questions as follows:
Thanks for your suggestion. Following it, we have extended our evaluation to measure false-positive rates (FPR) on neighboring retained classes across 16 selective forgetting requests, including comparisons with additional baselines such as Boundary Shrink, SCAR, and DIET (more detail in Appendix O, Tables 37–38).
Following our Near-Retain evaluation, neighboring classes are defined as the three nearest retained classes of each forgotten class. Each image is classified over the full label set, including forgotten classes. We calculate FPR for each neighboring class and average across classes and requests. We report FPR on retained images only, to assess changes to retained-class boundaries, and on all images, including forgotten ones.
False-positive analysis averaged over 16 selective forgetting requests (%, three-seed means).
The results show that LIFT reduces Forget accuracy from 71.6% to 1.5% while preserving neighboring-class recall (65.8% versus 65.5% for Original CLIP). Importantly, FPR on retained images remains close to Original CLIP (0.88% versus 0.77%), with only minor changes on 15 of the 16 requests. EuroSAT shows the largest increase (7.68% to 9.12%), as detailed in Appendix O (Table 37).
When forgotten images are included, FPR increases from 0.74% to 1.03%, partly because some forgotten images are reassigned to neighboring classes (27.3% versus 10.5% for Original CLIP). However, this increase is much smaller when considering retained images only, indicating that LIFT largely preserves neighboring-class recognition.
Compared with the baselines, LIFT is the only evaluated method achieving low average Forget accuracy while maintaining neighboring-class recall close to Original CLIP. Other methods with strong forgetting preserve only 2.7–32.0% neighboring-class recall, whereas DAMP maintains recall but leaves 48.3% Forget accuracy.
These findings demonstrate that LIFT substantially reduces target recognition while largely preserving neighboring-class boundaries, with only a limited increase in false positives on retained images. We have added this analysis to Section 6 (paragraph "False positives on neighboring classes") and the per-request results to Appendix O.