Forgotten or Just Unseen? Semantic Erasure and Auditing for Face Unlearning.

In open-set face recognition, forgetting an identity and never having seen it in the first place can exhibit indistinguishable behavior, raising a fundamental question of what it truly means to “unlearn” an identity. We term this the Forgetting-Or-Generalization (FOG) ambiguity. Existing unlearning measures rely on accuracy-based metrics, whereas face recognition operates in an open-set, verification-based setting where unseen identities are expected to be recognized correctly. This mismatch makes facial erasure difficult to evaluate and renders verification-accuracy-based unlearning metrics misleading. We formalize this facial unlearning problem and introduce a generative auditing framework for semantic visualization of identity erasure, revealing that existing unlearning methods retain identity information despite appearing successful under conventional metrics. To address this, we propose a feature-space unlearning algorithm that removes structural identity groupings by erasing identity regions using region-aware surrogate mechanisms while preserving recognition of non-target identities. Our framework shifts face unlearning evaluation from accuracy-driven to semantics-based assessment, providing stronger empirical privacy validation. Experiments on CASIA-WebFace, WebFace4M, MS1Mv3, and D-LORD show that FUSE achieves the lowest forget-set semantic residual among nine unlearning methods (as low as 0.27), while preserving retain-set verification and open-set accuracy (99.31% on LFW, against 99.53% for the original model).