Task-Agnostic Domain Unlearning for Vision-Language Models via Contrastive Editing

Vision-Language Models (VLMs) such as CLIP acquire domain-general image–text alignments, allowing the same semantic concept to be accessed across visual domains. This behavior becomes undesirable when a semantic association should be suppressed only in selected domains while remaining usable elsewhere. Existing domain unlearning approaches for CLIP are primarily evaluated through classification behavior and feature-level domain separation, which need not directly control the cross-modal ranking geometry used by both prompt-based classification and retrieval. We introduce Domain-Aware Contrastive Editing (DACE), a framework for approximate domain unlearning that directly edits domain-conditioned image–text relations. DACE preserves alignment on retain domains and applies one of three forgetting operators on forget domains: absolute similarity suppression (DACE-S), mismatch reassignment (DACE-M), or hard-negative margin reversal (DACE-H). We formalize the connection between alignment editing and downstream forgetting through cross-modal ranking margins: zero-shot classification and Recall@$K$ are characterized by order statistics of the same similarity geometry, and the DACE-H hinge objective upper-bounds residual top-1 success on forget samples. This makes the edit task-independent at the level of the shared similarity geometry, with formal guarantees for prompt-based classification and top-$K$ retrieval, without implying certified information removal. Across Office-Home, Mini DomainNet, and DomainNet, DACE suppresses forget-domain behavior while preserving retain-domain utility across classification and bidirectional retrieval with both CLIP and SigLIP. Representation and relearning analyses further indicate that behavioral forgetting can result from disrupting cross-modal accessibility without necessarily erasing all class-discriminative visual information.