Learning to Forget Undesired Anomalies: Category Unlearning for Weakly Supervised Video Anomaly Detection

Weakly supervised video anomaly detection (WSVAD) localizes anomalous events using only video-level labels. However, models trained under this paradigm are often deployed in various real-world scenarios, where certain categories previously treated as anomalies may no longer be considered anomalous. This issue is particularly challenging because WSVAD lacks frame-level category supervision, while many recent state-of-the-art methods couple a class-agnostic binary branch with a fine-grained classification branch. Naively masking a category output suppresses only its fine-grained prediction, leaving the shared binary anomaly evidence largely unaffected. To address this limitation, we propose Category-Conditioned Dual-Branch Modulation (C-DBM), a lightweight adapter that enables on-demand unlearning of specific anomaly categories for frozen WSVAD detectors. Given a set of retained categories, C-DBM employs category-conditioned cross-attention to extract category-specific temporal information and modulates both the fine-grained representations and the binary anomaly logits accordingly. After a one-time adapter training stage, different unlearning requests can be satisfied simply by updating the retained-category state, without any request-time optimization or access to the original training data. Experiments across four base-model–dataset combinations show that C-DBM supports both single- and multi-category unlearning and substantially reduces residual fine-grained responses for forgotten categories. Moreover, compared with other unlearning baselines, C-DBM more effectively suppresses forgotten-category predictions, while preserving both binary and fine-grained detection performance under the relabeled full-test protocol. Overall, C-DBM demonstrates effective anomaly-category unlearning while maintaining performance across both branches of modern WSVAD frameworks.