- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies concept unlearning in CLIP-style VLMs. It argues that source-domain forget accuracy hides three failure modes:
- Visual bias: recognition survives in unseen visual domains.
- Association leakage: alternative expressions recover the concept.
- Semantic collapse: post-edit predictions become arbitrary, or neighboring classes are damaged.
The proposed CSOA trains a rank-16 LoRA update (20,480 parameters) on the visual projection of a frozen CLIP ViT-B/32. It uses four sets of text anchors (erase / control / peer / reject, called "QAS") and four losses:
- moves the edited forget-image features away from their original features and from the erase-anchor mean .
- preserves retain-image features.
- pulls forget-image features toward the text embedding of a user-declared ancestor (e.g., apple fruit).
- preserves a softmax over peer and context anchors.
Training also uses a forget/retain gradient projection (PCGrad-style) and a three-stage curriculum. A variant, CSOA-R, adds class-level distillation, replay of images from the same coarse class, and a lower control weight.
The main evaluation edits 10 CIFAR-100 concepts one at a time, with 5 seeds. It tests on CIFAR-100, ImageNet and frozen DomainNet (real/sketch/clipart), and adds held-out description queries, an ancestor-preference score (HCS) and exact-label accuracy on classes from the same coarse category ("peers"). Comparisons include GA, NegGrad+, SSD, SalUn, CLIPErase, CCUP, SLUG, Text-image Space and a knowledge-matched control. Extensions cover two more CLIP backbones, 40 additional concepts, membership-inference audits and adapted generative comparisons on Qwen2.5-VL and LLaVA-1.5.
The paper reports that CSOA reaches near-zero target recognition with a low description residual at a moderate retain cost, and that CSOA-R further reduces retain and peer losses.
Critical strengths
1. A useful joint evaluation. The paper evaluates source forgetting, cross-domain transfer, held-out descriptions and the post-edit destination on the same edited model. It shows concrete cases where low source forget accuracy hides residual recognition:
- CCUP reaches 0% CIFAR target accuracy but recognizes the target from 28.38% of held-out descriptions.
- GA reaches 0% on CIFAR but still recognizes 46.32% of the ImageNet target images.
This is a valuable message for how unlearning in VLMs should be reported.
2. Transparent and extensive experimentation. The paper uses a fixed final checkpoint, per-concept tables, a full factorial study of CSOA-R (three factors, eight configurations) and a knowledge-matched control. It also reports negative results openly, such as the failure of the validation-selected configuration on ImageNet and the higher membership-inference AUC in the 25-image setting. This makes the claims checkable and gives the authors a solid basis for addressing the concerns below.
Critical weaknesses
1. The gains are not yet clearly attributed to the proposed components, and HCS is closely tied to the training objective.
The paper presents QAS and the anchor-based direction as the main remedies for association leakage and visual bias. The authors' own ablations, however, point elsewhere:
- Removing the directional penalty leaves all forgetting metrics unchanged (Appendix C.6).
- A canonical-only erase bank changes the description residual only from 1.10% to 1.38% (Table 6).
- Removing raises the ImageNet residual from 0.07% to 26.01%.
- Removing the gradient projection raises retain loss from 1.06 to 6.23 points.
So the effect seems to come mainly from the ancestor-pull loss and the gradient projection.
In addition, HCS is computed against the same ancestor vector that optimizes: is the mean of over forget images. A high HCS is therefore expected by construction, and on its own it is weak evidence of semantic redirection.
I believe this is addressable. A re-evaluation of the saved checkpoints with independent ancestor phrasings (Q2), together with a clearer statement of which components carry the effect, would substantially strengthen the paper.
2. The comparisons do not yet isolate CSOA's contribution.
- Several baselines barely change the model. SLUG selects a zero edit in all runs; SSD keeps 69.04% target accuracy; MMUnlearner in the LLaVA study stays at 52.50% against 55.00% for the original model.
- External baselines receive only the class name, while CSOA uses the full anchor banks.
- The knowledge-matched control omits the gradient projection that, according to the ablation, accounts for much of CSOA's selectivity.
- On the standard metrics, the training-free CCUP has lower CIFAR-retain, ImageNet-retain and peer loss than CSOA (0.29 / 1.54 / 3.43 vs. 1.10 / 3.34 / 8.92 points). CSOA's clear advantage is the description residual, and that gap may close if CCUP is given the same erase bank.
Two inexpensive controls (Q1) would show whether CSOA's advantage holds once knowledge and the optimization trick are matched.
Questions for authors
I have limited my questions to the points that would most change my assessment. All of them can be answered with a closed-form computation, a re-evaluation of the saved checkpoints, or short training runs on cached features. If Q1 and Q2 are answered convincingly, I am willing to raise my score.
Q1. Matched controls (main factor in my rating).
- (a) What are CCUP's results when is the same 15-string erase bank? This is closed-form and needs no training.
- (b) What are the results of the knowledge-matched control trained with the same forget/retain gradient projection?
- If CSOA keeps a clear advantage over both in retain loss, peer loss or description residual, I would view the method contribution much more favorably.
- (c) Clarification: the supplementary code includes an implementation of RUCLIP (CLIP concept unlearning with semantic anchors, ESWA 2026). If results exist, please include them, or explain why they were left out. Either way, a short discussion of how CSOA differs from RUCLIP would help position the work.
Q2. HCS with an independent ancestor vector (re-evaluation only). What is HCS when the ancestor vector is built from phrasings or templates not used to construct during training, such as alternative hypernym wordings? If HCS stays high, the semantic-redirection claim is well supported.
Q3. Recoverability (re-evaluation only). On the saved edited features , what target accuracy does a linear probe or kNN classifier trained on a small number of labeled images achieve? This directly concerns the paper's motivation that recognition "may remain recoverable".
- Low recoverability would strengthen the paper considerably.
- High recoverability would also be acceptable to me, as long as the method is clearly framed as behavioral suppression.
Q4. Clarifications from existing records.
- (a) The code labels five concepts as SEEN (used for development) and five as HELDOUT. Could you report the main results separately for the two groups?
- (b) CSOA-R's peer replay samples from the same four same-coarse classes that the peer-loss metric evaluates. Could you confirm this and qualify the neighboring-class claim accordingly?
Suggestions (these do not affect my rating):
- In the generative section, please report the open and coarse prompt results in the main text alongside the direct prompt. On Qwen sketch/clipart and on LLaVA, GA+KL is competitive or better under those prompts.
- Consider moving the unused router and out of the main method description.
- Related work could add anchor-concept redirection in diffusion models (Kumari et al., ICCV 2023; UCE, WACV 2024) and linear concept erasure (LEACE, NeurIPS 2023).
- A brief ethics note on erasing demographic categories (the "woman" concept) would be helpful.
- Multi-concept or sequential edits and identity-level tasks would be natural future work.