acceptodds
Withdrawn from ICLR 2027

GrantFace: Who Gets to Recognize Your Face?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Facial data submitted to a legitimate recognition service can be reused by unauthorized face recognition systems. Our intuition is that an image contains much more than a recognizer uses: variations it largely ignores may still matter to other recognizers. We turn this model-specific insensitivity into protection spaces that preserve authorized matching while disrupting unauthorized recognition. Our method, GrantFace, constructs these low-sensitivity spaces by aggregating the authorized recognizer's input-to-embedding sensitivities over synthetic probes. The protection space is constructed once and generalizes across identities and inputs without modifying the authorized recognizer or accessing unauthorized recognizers. We prove leading-order optimality for authorized release agreement on synthetic probes and give sufficient conditions for a selective protection interval. Across four benchmarks, GrantFace retains an average authorized TAR of 93.29% while reducing unauthorized TAR to 4.41%, with selectivity persisting for recognizers sharing the same backbone and after adaptive fine-tuning. By eliminating per-input optimization, GrantFace protects a new face in 0.26 ms.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Face Recognition, Facial Privacy, Model-Specific Authorization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Q5hEOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines a relatively new problem to modify the face image so that it can be recognised only by an authorised face recognition model. The protected face is then evaluated against other unauthorised face recognition models where the performance drops. The protected images have a lot of changes (examples in Fig 3) that is also difficult to recognise for human.

Critical strengths
  • The paper proposes a new method for protecting face images to be recognisable by an authorised face recognition model.
  • Experiments are conducted with different model architectures and on different datasets.
Critical weaknesses
  • The main issue is with the definition of the task (which is new). The protected images have a lot of changes (examples in Fig 3) that is also difficult to recognise for human. This causes a lot of limitations in application. It is more interesting if the image remains recognisable for human.
  • The paper lacks baseline. If we assume that it is not important that the protected face can be recognised by human a gradient assent based approach on the identity embedding/class will be easily provide protection.
  • If the image cannot be recognised by human why should users send their protected image, instead they can send an encrypted image.
Questions for authors
  • Given that we are assuming that there is an authorised face recognition model, can we use a secret key to protect the image?
  • Assuming whitebox/blackbox access to the authorised face recognition model, can an adversary reconstruct the face image (to be recognised by other models) from protected image?
Reviewer_6PGhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to transform face inputs so that a designated authorized recognizer can continue matching identities while reducing the recognition capability of other recognizers. GRANTFACE aggregates the authorized model’s input-to-normalized-embedding Jacobian Gram matrices over identity-independent random-noise probes, selects a low-sensitivity eigenspace, and adds random perturbations sampled from this space to each image. Once constructed, the space can be reused across users and inputs without per-image optimization or modifications to the recognition model. The paper proves leading-order optimality for agreement between protected releases of synthetic probes and derives a selective protection interval under explicit conditions. Experiments cover four face verification benchmarks, different recognizers, configurations sharing the same backbone, and fine-tuning and reconstruction attacks.

Critical strengths

I find the paper' s most interesting contribution to be its shift from optimizing perturbations for individual images to constructing a shared, model-level protection space. The method exploits the authorized recognizer' s insensitivity to certain input variations by aggregating input-to-embedding sensitivities over synthetic probes and identifying directions that have little effect on its output. Perturbations are then sampled within the resulting subspace to preserve authorized matching while disrupting other recognizers. What makes this idea appealing is that the protection space needs to be constructed offline only once for the authorized model and can subsequently be applied across identities and new inputs, without repeated per-image optimization or modifications to the authorized recognizer. Table 2 reports an average online processing time of just 0.261 ms per image, allowing the one-time construction cost to be amortized over many inputs. I find this design, which builds on the model' s own sensitivity structure and moves the main computational burden offline, both intuitive and clever.

Critical weaknesses
  1. I believe the paper needs to justify its application scenarios and practical value more concretely. The goal is to preserve the authorized model' s recognition capability while making protected faces difficult for other models to recognize. Although Section 2.2 describes internal processing by the service and third-party access to protected images, I remain unclear about how protection tied to a specific recognition model corresponds to real-world authorization of data use. In particular, facial data misuse can involve either direct recognition or collecting images to train new systems, and these are distinct risks. While the paper evaluates fine-tuning attacks, the results do not establish that protected images cannot be used for training. I would like the authors to provide a more concrete deployment workflow explaining which parties possess the original and protected images, how third parties obtain these data, and why restricting cross-model recognition alone provides an independent and meaningful privacy benefit in that setting. This would directly affect my assessment of the method' s practical applicability and the significance of the work.
  2. I still have concerns about the conditions under which cross-model protection holds and the limits of its generalization. My understanding is that GRANTFACE first constructs a low-sensitivity subspace for the authorized model and then achieves selective protection by exploiting other models' different responses within that subspace. Suppressing unauthorized recognition therefore depends on a sufficient sensitivity difference between the authorized and unauthorized models. In my view, the transferability observed in the paper may arise because the evaluated models have sufficiently different low-sensitivity subspaces, allowing the perturbations to affect unauthorized models. I believe the authors should discuss this transferability further, particularly because protection may fail to transfer when the unauthorized and authorized models have overlapping or similar low-sensitivity subspaces.
Questions for authors
  1. Could you provide a concrete real-world application scenario explaining which parties hold the original and protected images, how unauthorized parties obtain the protected images, and what practical privacy problem is addressed by allowing a designated model to recognize faces while restricting recognition by other models?
  2. If the unauthorized and authorized models have highly overlapping low-sensitivity subspaces, can GRANTFACE still achieve selective protection? Could you discuss its effectiveness and failure conditions in this case using existing theory or experiments, or provide evidence explaining why different recognition models in practice are unlikely to exhibit enough subspace overlap to substantially weaken protection?
Reviewer_rpRtOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GrantFace, a selective face protection technique that maintains matching for authorized face recognizers while suppressing recognition for other recognizers. A low-sensitivity subspace is constructed only once by aggregating the input-to-embedding sensitivity of the authorized recognizer against a synthetic noise probe, and each face is protected by adding random noise sampled from within this subspace.

Critical strengths

The attempt to link recognizer-specific insensitivity to a shared selective protection mechanism without per-sample optimization is interesting. The idea of ​​constructing a subspace only once using an identity-independent probe and leveraging the observation that this subspace is sensitive to other recognizers is simple yet unique.

Critical weaknesses
  • It is unclear what threat the proposed protection mitigates. Since users provide the raw images only to a trusted service and the protection is applied within that service, the protection is meaningful only when the protected image leaves the trusted service. However, that scenario is not clarified. This contrasts with comparative and cited selective protection studies and PPFR studies, most of which apply protection before the original is delivered to an untrusted party. (If the goal is to protect data stored by the service, practical deployments typically store features rather than images (e.g., Azure), and protecting such stored features is precisely the problem addressed by biometric template protection.) Furthermore, since security relies on the confidentiality of a single BAB_A per model shared by all users, it is also necessary to explain how the protection is updated when the BAB_A is exposed.

  • The paper shows how attention and embeddings change after protection, but does not explain why a subspace that barely affects the authorized recognizer strongly affects other recognizers. The theoretical conditions for selectivity are not expressed in terms of interpretable properties of the recognizers, and thus offer little insight into when selectivity arises. It also remains unclear what input structure the protection subspace captures, and why sensitivity measured on synthetic noise probes transfers to real face images.

  • The security is supported on an assumption that is not justified. As the authors acknowledge, once the protection subspace is known, the noise can be removed by a simple projection, so security depends entirely on the assumption that this subspace cannot be recovered. However, no argument or evidence is given for this assumption.

  • Robustness evaluation is limited. Attacks known to undermine fixed protective perturbations, such as perturbation detection or adversarial training, etc.. and common image preprocessing can affect on perturbation effect such as JPEG compression or blurring are not evaluated.

Questions for authors
  • In what scenario would (protected) images reach an untrusted party? What advantage does the proposed method offer over applying template protection to stored features, and can protection be renewed if BAB_A is exposed?

  • What makes the protection subspace insensitive for the authorized recognizer but not for others? Could the authors provide an analysis of what this subspace captures and under what conditions cross-model selectivity holds?

  • What justifies the assumption that the BAB_A cannot be recovered, and would protection effect hold against attacks exploiting the shared subspace or under common preprocessing?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.