acceptodds
Under review as a conference paper at ICLR 2027

CSOA: Concept-Selective Adaptation with Semantic Control for Vision-Language Unlearning

Abstract

Concept unlearning in vision-language models (VLMs) is increasingly critical for selective removal of unwanted semantic information. However, practice demonstrates that while existing methods report success on traditional metrics, target recognition may remain recoverable through unconventional queries or visual domains. We identify three critical failure modes: visual bias arises when unlearning on narrow forget sets leaves recognition intact in unseen visual domains; association leakage occurs when suppressing specific text templates leaves alternative expressions as retrieval backdoors; and semantic collapse results when unconstrained adaptation disrupts meaningful post-unlearning predictions and neighboring concepts. To address these challenges, we introduce CSOA, a concept-selective adaptation framework with semantic control. This approach combines a quadrant anchor system specifying the linguistic neighborhood with hierarchical control that guides edited representations toward user-specified semantic ancestors, while preserving non-target utility. Knowledge-matched controls and factorial interventions assess semantic redirection and the retention-focused variant CSOA-R. Tests on three CLIP backbones and 40 additional concepts examine breadth; paired generative comparisons distinguish recognition suppression from answer disclosure.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine unlearning, concept unlearning, vision-language models, semantic control, cross-domain evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_t6xaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies concept unlearning in CLIP-style VLMs. It argues that source-domain forget accuracy hides three failure modes:

  • Visual bias: recognition survives in unseen visual domains.
  • Association leakage: alternative expressions recover the concept.
  • Semantic collapse: post-edit predictions become arbitrary, or neighboring classes are damaged.

The proposed CSOA trains a rank-16 LoRA update ΔW=BA\Delta W = BA (20,480 parameters) on the visual projection of a frozen CLIP ViT-B/32. It uses four sets of text anchors (erase / control / peer / reject, called "QAS") and four losses:

  • Lerase\mathcal{L}_{\mathrm{erase}} moves the edited forget-image features z^\hat{z} away from their original features zorigz_{\mathrm{orig}} and from the erase-anchor mean tˉerase\bar{t}_{\mathrm{erase}}.
  • Lretain\mathcal{L}_{\mathrm{retain}} preserves retain-image features.
  • Lctrl\mathcal{L}_{\mathrm{ctrl}} pulls forget-image features toward the text embedding tˉctrl\bar{t}_{\mathrm{ctrl}} of a user-declared ancestor (e.g., apple →\rightarrow fruit).
  • Lstruct\mathcal{L}_{\mathrm{struct}} preserves a softmax over peer and context anchors.

Training also uses a forget/retain gradient projection (PCGrad-style) and a three-stage curriculum. A variant, CSOA-R, adds class-level distillation, replay of images from the same coarse class, and a lower control weight.

The main evaluation edits 10 CIFAR-100 concepts one at a time, with 5 seeds. It tests on CIFAR-100, ImageNet and frozen DomainNet (real/sketch/clipart), and adds held-out description queries, an ancestor-preference score (HCS) and exact-label accuracy on classes from the same coarse category ("peers"). Comparisons include GA, NegGrad+, SSD, SalUn, CLIPErase, CCUP, SLUG, Text-image Space and a knowledge-matched control. Extensions cover two more CLIP backbones, 40 additional concepts, membership-inference audits and adapted generative comparisons on Qwen2.5-VL and LLaVA-1.5.

The paper reports that CSOA reaches near-zero target recognition with a low description residual at a moderate retain cost, and that CSOA-R further reduces retain and peer losses.

Critical strengths

1. A useful joint evaluation. The paper evaluates source forgetting, cross-domain transfer, held-out descriptions and the post-edit destination on the same edited model. It shows concrete cases where low source forget accuracy hides residual recognition:

  • CCUP reaches 0% CIFAR target accuracy but recognizes the target from 28.38% of held-out descriptions.
  • GA reaches 0% on CIFAR but still recognizes 46.32% of the ImageNet target images.

This is a valuable message for how unlearning in VLMs should be reported.

2. Transparent and extensive experimentation. The paper uses a fixed final checkpoint, per-concept tables, a full factorial study of CSOA-R (three factors, eight configurations) and a knowledge-matched control. It also reports negative results openly, such as the failure of the validation-selected configuration on ImageNet and the higher membership-inference AUC in the 25-image setting. This makes the claims checkable and gives the authors a solid basis for addressing the concerns below.

Critical weaknesses

1. The gains are not yet clearly attributed to the proposed components, and HCS is closely tied to the training objective.

The paper presents QAS and the anchor-based direction as the main remedies for association leakage and visual bias. The authors' own ablations, however, point elsewhere:

  • Removing the directional penalty Ldir\mathcal{L}_{\mathrm{dir}} leaves all forgetting metrics unchanged (Appendix C.6).
  • A canonical-only erase bank changes the description residual only from 1.10% to 1.38% (Table 6).
  • Removing Lctrl\mathcal{L}_{\mathrm{ctrl}} raises the ImageNet residual from 0.07% to 26.01%.
  • Removing the gradient projection raises retain loss from 1.06 to 6.23 points.

So the effect seems to come mainly from the ancestor-pull loss and the gradient projection.

In addition, HCS is computed against the same ancestor vector tˉctrl\bar{t}_{\mathrm{ctrl}} that Lctrl\mathcal{L}_{\mathrm{ctrl}} optimizes: Lctrl\mathcal{L}_{\mathrm{ctrl}} is the mean of 1−cos⁡(z^,tˉctrl)1-\cos(\hat{z}, \bar{t}_{\mathrm{ctrl}}) over forget images. A high HCS is therefore expected by construction, and on its own it is weak evidence of semantic redirection.

I believe this is addressable. A re-evaluation of the saved checkpoints with independent ancestor phrasings (Q2), together with a clearer statement of which components carry the effect, would substantially strengthen the paper.

2. The comparisons do not yet isolate CSOA's contribution.

  • Several baselines barely change the model. SLUG selects a zero edit in all runs; SSD keeps 69.04% target accuracy; MMUnlearner in the LLaVA study stays at 52.50% against 55.00% for the original model.
  • External baselines receive only the class name, while CSOA uses the full anchor banks.
  • The knowledge-matched control omits the gradient projection that, according to the ablation, accounts for much of CSOA's selectivity.
  • On the standard metrics, the training-free CCUP has lower CIFAR-retain, ImageNet-retain and peer loss than CSOA (0.29 / 1.54 / 3.43 vs. 1.10 / 3.34 / 8.92 points). CSOA's clear advantage is the description residual, and that gap may close if CCUP is given the same erase bank.

Two inexpensive controls (Q1) would show whether CSOA's advantage holds once knowledge and the optimization trick are matched.

Questions for authors

I have limited my questions to the points that would most change my assessment. All of them can be answered with a closed-form computation, a re-evaluation of the saved checkpoints, or short training runs on cached features. If Q1 and Q2 are answered convincingly, I am willing to raise my score.

Q1. Matched controls (main factor in my rating).

  • (a) What are CCUP's results when TfT_f is the same 15-string erase bank? This is closed-form and needs no training.
  • (b) What are the results of the knowledge-matched control trained with the same forget/retain gradient projection?
  • If CSOA keeps a clear advantage over both in retain loss, peer loss or description residual, I would view the method contribution much more favorably.
  • (c) Clarification: the supplementary code includes an implementation of RUCLIP (CLIP concept unlearning with semantic anchors, ESWA 2026). If results exist, please include them, or explain why they were left out. Either way, a short discussion of how CSOA differs from RUCLIP would help position the work.

Q2. HCS with an independent ancestor vector (re-evaluation only). What is HCS when the ancestor vector is built from phrasings or templates not used to construct tˉctrl\bar{t}_{\mathrm{ctrl}} during training, such as alternative hypernym wordings? If HCS stays high, the semantic-redirection claim is well supported.

Q3. Recoverability (re-evaluation only). On the saved edited features z^\hat{z}, what target accuracy does a linear probe or kNN classifier trained on a small number of labeled images achieve? This directly concerns the paper's motivation that recognition "may remain recoverable".

  • Low recoverability would strengthen the paper considerably.
  • High recoverability would also be acceptable to me, as long as the method is clearly framed as behavioral suppression.

Q4. Clarifications from existing records.

  • (a) The code labels five concepts as SEEN (used for development) and five as HELDOUT. Could you report the main results separately for the two groups?
  • (b) CSOA-R's peer replay samples from the same four same-coarse classes that the peer-loss metric evaluates. Could you confirm this and qualify the neighboring-class claim accordingly?

Suggestions (these do not affect my rating):

  • In the generative section, please report the open and coarse prompt results in the main text alongside the direct prompt. On Qwen sketch/clipart and on LLaVA, GA+KL is competitive or better under those prompts.
  • Consider moving the unused router and Ldir\mathcal{L}_{\mathrm{dir}} out of the main method description.
  • Related work could add anchor-concept redirection in diffusion models (Kumari et al., ICCV 2023; UCE, WACV 2024) and linear concept erasure (LEACE, NeurIPS 2023).
  • A brief ethics note on erasing demographic categories (the "woman" concept) would be helpful.
  • Multi-concept or sequential edits and identity-level tasks would be natural future work.
Reviewer_YeJcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

This paper studies concept unlearning in vision-language models through three failure modes: cross-domain visual bias, residual associations under alternative language expressions, and semantic collapse. It introduces CSOA, combining semantic anchor sets, hierarchical redirection, and retention-aware low-rank adaptation, together with CSOA-R for improved retained utility. The evaluation includes several CLIP backbones, cross-domain tests, component studies, and additional generative-model diagnostics.

Critical strengths

Strengths

The paper identifies practically meaningful evaluation gaps beyond standard forget-set accuracy, particularly alternative expressions and post-unlearning semantic behavior.

The evaluation is extensive, including a knowledge-matched control, explicit component interventions, three CLIP backbones, additional concepts, and cross-style generalization tests.

The distinction between the original CSOA recipe and the retention-oriented CSOA-R variant is clearly reported rather than presenting one operating point as uniformly better.

Critical weaknesses

Weaknesses

The hierarchical-control metric is closely aligned with the training objective. Table 3 reports 98.72% HCS for CSOA, but the metric evaluates preference for the same declared ancestor used during training. A small independent analysis of semantic plausibility or sibling-class confusion would strengthen the claim that the redirection remains meaningful beyond the chosen metric.

There is a noticeable trade-off between suppression and neighboring-class utility. In Table 1, CSOA has an 8.92-point peer accuracy loss, reduced to 0.37 points by CSOA-R at the cost of somewhat higher target residuals. A clearer discussion of the operating conditions under which each variant should be preferred would improve the practical interpretation.

The distinction between behavioral suppression and information removal deserves emphasis. Section 4.3 reports membership-inference AUCs of 0.654 and 0.691 with only 25 forget images, and the generative diagnostics report residual open-description disclosure. These results suggest that the strongest interpretation of the method is selective behavioral editing rather than complete semantic erasure.

Questions for authors

Questions

How sensitive are the results to the choice of semantic ancestor and to cases with multiple plausible ancestors?

Could the authors report a small set of representative examples where target-label suppression succeeds but related-concept predictions or open-ended disclosures remain problematic?

Reviewer_7buPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the concept unlearning problem in VLMs and proposes a concept-selective adaptation framework with semantic control (CSOA) to tackle this problem. CSOA adopts four groups of semantic anchors to specify what should be suppressed, where predictions may be redirected, and which semantics should be preserved. This paper further develops CSOA-R with retention distillation, peer replay, and moderated semantic control to preserve neighboring-class utility. The authors conduct evaluations on two CLIP backbones, 40 additional concepts, and paired generative comparisons. The results suggest that CSOA can suppress recognition under several evaluation conditions, while CSOA-R improves neighboring-class retention, demonstrating the effectiveness to some extent.

Critical strengths
  1. This paper provides a broader evaluation compared with previous concept unlearning methods, which jointly evaluates cross-domain recognition, unseen linguistic descriptions, semantic destinations, and neighboring-class preservation. The paired generative diagnostics further distinguish reduced affirmative responses from actual suppression of target disclosure.
  2. CSOA explicitly specifies acceptable semantic ancestors and protects neighboring concepts, which makes the intended post-unlearning behavior more precise. CSOA-R further improves local retention. Experimental results demonstrate the effectiveness of both CSOA and CSOA-R.
Critical weaknesses
  1. CSOA combines established techniques, including semantic anchoring, ancestor alignment, low-rank adaptation, retention distillation, replay, and conflict-aware gradient projection. Although such a combination seems effective and reasonable, the experiments more clearly validate the effectiveness of the overall framework, whereas the model with isolated or single proposed components is not fully investigated.
  2. CSOA depends on predefined anchor banks to specify target expressions, permitted semantic destinations, and neighboring concepts to preserve. Although held-out descriptions and the 40-concept extension support generalization, sensitivity to anchor coverage and composition remains underexplored. It is unclear whether selective forgetting remains reliable when target expressions are incompletely specified or neighboring concepts overlap semantically.
Questions for authors
  1. The authors should provide more ablations to demonstrate the gains from isolated components, rather than only removing single components but combining others.
  2. How sensitive is CSOA to incomplete target-expression coverage or semantic overlap with preserved concepts?
  3. How to distinguish semantic redirection toward ancestors from removal of target-specific information? Whether it involves reduced accessibility under the evaluation prompts or more information loss.
  4. Are the generative experiments evidence of successful method transfer or primarily diagnostics of response-level metrics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.