Keep Identity, Hide Attributes: Spectral Subspaces for Selective Facial Privacy

Face images used for identity verification inevitably expose soft biometric attributes such as gender, age, and facial expression, enabling downstream models to infer information that users never intended to reveal. Selectively protecting these attributes remains difficult. Generative editing methods often modify identity-relevant facial structure, whereas adversarial perturbations distribute changes across the entire image and unintentionally affect attributes that users wish to preserve. We ask a simple question: if an attribute classifier relies on only a few directions in image space, why perturb the whole face? We propose GAUSS (Geometric Attribute Suppression via Spectral Subspaces) answers this by constructing perturbations within low-dimensional, attribute-specific spectral subspaces learned from classifier input gradients. An instance-adaptive coefficient network then generates image-specific perturbations, allowing a single model to protect arbitrary combinations of user-selected attributes without a generative backbone. Beyond the method itself, we show that attribute subspaces are largely classifier-specific and that language-defined semantic supervision substantially improves transfer across architecture families. On CelebA-HQ, GAUSS suppresses each target attribute in at least 92% of test images, achieves the highest non-target retention across all attribute pairs among eight baselines, preserves face verification performance with 71$\times$ fewer trainable parameters, and transfers to unseen classifiers at more than twice the rate of prior generative approaches while maintaining 35.79 dB PSNR.