GrantFace: Who Gets to Recognize Your Face?

Facial data submitted to a legitimate recognition service can be reused by unauthorized face recognition systems. Our intuition is that an image contains much more than a recognizer uses: variations it largely ignores may still matter to other recognizers. We turn this model-specific insensitivity into protection spaces that preserve authorized matching while disrupting unauthorized recognition. Our method, GrantFace, constructs these low-sensitivity spaces by aggregating the authorized recognizer's input-to-embedding sensitivities over synthetic probes. The protection space is constructed once and generalizes across identities and inputs without modifying the authorized recognizer or accessing unauthorized recognizers. We prove leading-order optimality for authorized release agreement on synthetic probes and give sufficient conditions for a selective protection interval. Across four benchmarks, GrantFace retains an average authorized TAR of 93.29% while reducing unauthorized TAR to 4.41%, with selectivity persisting for recognizers sharing the same backbone and after adaptive fine-tuning. By eliminating per-input optimization, GrantFace protects a new face in 0.26 ms.