InvFace: Privacy-Preserving Face Recognition Using Identity-Distilling Inversion

Sharing unprotected images for face recognition can expose sensitive visual information beyond what is needed for identity comparison. Existing protection methods transform face images into identity-preserving and visual-obscuring representations, yet their effectiveness is validated mainly empirically, leaving privacy disclosure unquantified. This paper proposes a stepwise paradigm that decomposes monolithic protection into explicit, atomic transitions whose cumulative effects are auditable. It is instantiated as a novel protection method, InvFace. InvFace performs identity-distilling inversion, where an identity-conditioned diffusion model, inverted without the identity condition, produces noise-like yet recognition-exploitable latents. This is followed by mirrored classifier-free guidance, which strengthens identity retention and disrupts direct recovery, and quantization, which makes the representation finite-valued. Taken together, this design admits information-theoretic bounds on visual disclosure beyond identity and generic reconstruction error. Extensive experiments show that InvFace matches leading prior works in recognition utility while providing stronger resistance to multiple reconstruction attacks.