Forget at Sight: Training-Free Domain Unlearning in Vision-Language Models

Vision-language models such as CLIP are designed to recognize the same object across different visual domains. This makes domain unlearning difficult: an intervention that suppresses recognition in one domain can also damage the same classes in other domains. Existing approaches rely on iterative optimization and access to retain-domain images, limiting their use when model updates are costly or retain data are unavailable. We introduce GATE-DU (Gated Activation Steering for Targeted Erasure in Domain Unlearning), a training-free framework that separates when to erase from how to erase. For each input, a domain gate first estimates whether it belongs to the forget domain. Only then does GATE-DU contract its representation toward a class-agnostic forget-domain centroid, suppressing class recognition while leaving retain-domain inputs unchanged. GATE-DU obtains its gating evidence in three ways. When retain images are unavailable, GATE-DU (T) compares the forget-domain description against known retain-domain descriptions using the frozen text branch. GATE-DU (O) starts from this text gate and progressively grounds it in visual evidence from the unlabeled evaluation stream, first through a class-paired prototype and then through domain-LDA. When labeled retain images are available, GATE-DU (V) directly constructs a class-paired domain-LDA gate and calibrates it using a closed-form Gaussian posterior. All variants keep the model, prompts, and erasure payload frozen, requiring no gradients, adapters, or replay. Experiments on PACS, OfficeHome, and Mini-DomainNet across class fractions \(\gamma\in{0.25,0.5,0.75,1.0}\) show that GATE-DU is competitive with or outperforms optimization-based methods, particularly in low-\(\gamma\) open-vocabulary settings. Our results highlight conditional intervention as a practical principle for domain unlearning: effective erasure requires deciding not only how to forget, but also when to forget.