acceptodds
Under review as a conference paper at ICLR 2027

Keep Identity, Hide Attributes: Spectral Subspaces for Selective Facial Privacy

Abstract

Face images used for identity verification inevitably expose soft biometric attributes such as gender, age, and facial expression, enabling downstream models to infer information that users never intended to reveal. Selectively protecting these attributes remains difficult. Generative editing methods often modify identity-relevant facial structure, whereas adversarial perturbations distribute changes across the entire image and unintentionally affect attributes that users wish to preserve. We ask a simple question: if an attribute classifier relies on only a few directions in image space, why perturb the whole face? We propose GAUSS (Geometric Attribute Suppression via Spectral Subspaces) answers this by constructing perturbations within low-dimensional, attribute-specific spectral subspaces learned from classifier input gradients. An instance-adaptive coefficient network then generates image-specific perturbations, allowing a single model to protect arbitrary combinations of user-selected attributes without a generative backbone. Beyond the method itself, we show that attribute subspaces are largely classifier-specific and that language-defined semantic supervision substantially improves transfer across architecture families. On CelebA-HQ, GAUSS suppresses each target attribute in at least 92% of test images, achieves the highest non-target retention across all attribute pairs among eight baselines, preserves face verification performance with 71 fewer trainable parameters, and transfers to unseen classifiers at more than twice the rate of prior generative approaches while maintaining 35.79 dB PSNR.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Facial privacy, soft biometric suppression, adversarial perturbation, spectral subspace learning, identity preservation, transferability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1XhiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GAUSS, a lightweight image perturbation framework for selective soft-biometric privacy protection. The method constructs low-dimensional Spectral Attribute Subspaces (SAS) through spectral decomposition of the input gradients of frozen attribute classifiers and generates attribute-specific protective perturbations within these subspaces. Building on this, GAUSS introduces an Instance-Adaptive Perturbation (IAP) module that predicts image-dependent subspace coefficients and spatial masks to selectively suppress different sensitive attributes. In addition, the authors employ joint supervision from multiple attribute classifiers and semantic directions derived from CLIP text embeddings to improve the transferability of the perturbations to unseen model architectures.

Critical strengths

The paper investigates the role of low-dimensional attribute-sensitive subspaces in selective perturbation generation by examining the spectral structure of attribute classifiers' input gradients. The experiments demonstrate that low-dimensional learnable subspaces can achieve high attribute prediction flip rates and that the initial spectral subspaces estimated from different data subsets exhibit a certain degree of stability. These findings provide a valuable representation-geometric perspective on selective adversarial perturbations.

Critical weaknesses
  1. The paper primarily uses Flip Rate to evaluate attribute privacy protection, measuring the proportion of protected images for which the classifier's prediction differs from the ground-truth attribute label. However, for binary sensitive attributes, a high prediction flip rate does not necessarily imply that the original attribute information has been effectively concealed. For example, if the protection mechanism consistently causes a binary attribute classifier to predict the opposite of the true label for the vast majority of samples, an attacker aware of this behavior could potentially recover the original attributes simply by inverting the classifier's predictions. This requires no classifier retraining. The paper reports a white-box prediction flip rate of 99.56% for the Young attribute, but this result alone does not establish a corresponding level of privacy protection. On the contrary, for the same binary predictions, a simple label-inversion strategy could potentially achieve a comparable attribute recovery rate. Therefore, the current experiments more directly demonstrate that attribute classifier predictions are disrupted, rather than that sensitive attribute information has been effectively removed from the protected images.

  2. GAUSS is fundamentally an adversarial perturbation generation method with a bounded perturbation magnitude, yet the paper primarily compares it against generative facial attribute editing and privacy-preserving methods. Although these approaches share similar application objectives, they differ substantially in how they modify images, their optimization objectives, generative priors, and computational complexity. Consequently, some of GAUSS's advantages in terms of PSNR, identity preservation, model size, and inference latency may arise from the inherent differences between localized pixel-level perturbations and generative image editing, rather than solely from the proposed spectral subspace design. Moreover, although the Related Work section discusses attribute-based adversarial perturbations, mutual information-based attacks, class-wise universal perturbations, and other structured perturbation methods, the main experiments lack systematic comparisons with these more closely related approaches.

Questions for authors
  1. The paper primarily uses the Flip Rate of binary attribute classifiers as a privacy protection metric. However, when classifier predictions are consistently flipped, an attacker aware of the protection mechanism may recover the original attributes simply by reversing the predictions. Could the authors explain how GAUSS ensures that sensitive attribute information is effectively concealed, rather than merely causing existing classifiers to make systematic prediction errors?

  2. Could the authors compare the proposed gradient-SVD initialization with a randomly initialized orthonormal (k)-dimensional subspace, while keeping the IAP module, training losses, training budget, and orthogonal refinement procedure identical? Additionally, I suggest reporting the principal-angle overlap between the initial spectral subspace and the final learned subspace to quantify the geometric changes during training.

  3. Could the authors compare GAUSS against strong adversarial attack methods, such as PGD, MI-FGSM, and DI-FGSM, as well as recent perturbation-based approaches for soft-biometric privacy, under the same perturbation budget and perceptual quality constraints?

  4. The experimental setup explicitly identifies DINOv2-L, SigLIP-B, and CLIP-H as held-out foundation-model evaluators. However, Table 3 reports protected-image results only for SigLIP-B and CLIP-H, without providing the corresponding results for DINOv2-L.

5 comments
AuthorsOpenReview ↗

Attribute Recovery via Prediction Inversion and DINOv2-L Evaluation

1. Attribute Prediction Inversion and Privacy Protection

We thank the reviewer for highlighting the possibility of recovering protected attributes through prediction inversion. While we acknowledge that reversing predictions may counteract protection under certain conditions, its effectiveness mostly depends on the adversary's knowledge and the consistency of prediction reversal across different models. In a white-box setting, where perturbations are specifically optimized against a known classifier, GAUSS achieves approximately 96% to 98% suppression. Under such conditions, an adversary who knows the protection mechanism and its systematic prediction-reversal behavior could potentially recover the original binary attributes through inversion. However, this assumption does not directly generalize to practical black-box scenarios, where the underlying classifiers, decision boundaries, and responses to protected images may differ substantially from those used during perturbation optimization. Importantly, our cross-model evaluations demonstrate that the prediction flip rates vary considerably across unseen architectures, typically ranging from approximately 50% to 75%. For binary attributes, indiscriminately reversing these predictions would yield only 50% to 75% recovery accuracy, substantially lower than the near-perfect recovery achievable in the white-box setting. Also, when the flip rate approaches 50%, prediction inversion provides little advantage over random guessing. Thus, successful inversion against a known classifier does not establish a universally reliable recovery strategy across heterogeneous inference systems. Also, real-world automated inference pipelines generally lack explicit information about whether a particular image has been protected, which attributes are selected for suppression, and whether the corresponding classifier predictions should be reversed. Applying inversion indiscriminately to both protected and unprotected images may introduce additional prediction errors. For multi-class attributes, the challenge becomes more complex because incorrect predictions do not necessarily correspond to a unique, reversible mapping between the original and predicted classes.

4. Missing DINOv2-L Evaluation

We have added the complete protected-image evaluation across all three held-out foundation models below.

Table R3C4: Attribute prediction flip rates (%) on held-out foundation models and perceptual quality.

MethodSigLIP-B ↑CLIP-H ↑DINOv2-L ↑Mean ↑PSNR (dB) ↑
PrivAD28.6028.4035.4730.8219.40
GAUSS (Ours)41.0026.7030.1332.6135.79

Although PrivAD achieves higher suppression on CLIP-H and DINOv2-L, GAUSS achieves a higher mean suppression rate across the three foundation-model evaluators (32.61% versus 30.82%). Importantly, GAUSS also maintains substantially higher visual fidelity (35.79 versus 19.40 dB PSNR). These findings demonstrate that GAUSS provides a favorable balance between average cross-model attribute suppression and preservation of the original facial appearance. We will include the missing DINOv2-L results in the revised manuscript.

AuthorsOpenReview ↗

Gradient-SVD Initialization and Principal-Angle Analysis and DINOv2-L Results

2. Gradient-SVD Initialization and Principal-Angle Analysis

To directly assess the contribution of gradient-SVD initialization, we conducted additional experiments comparing it with randomly initialized orthonormal subspaces of identical rank k=8, while maintaining the same IAP architecture, training objectives, perturbation budget, optimization schedule, and orthogonal refinement procedure. We further analyzed the geometric evolution of the learned subspaces using principal-angle measurements throughout the 40-epoch training.

Table R3C2: Comparison of random orthonormal and gradient-SVD initialization. Attribute suppression (%) and PSNR (dB); higher is better.

InitializationMale ↑Young ↑Smiling ↑Mean ↑PSNR (dB) ↑
Random orthonormal92.9295.4885.7491.3830.99
Gradient-SVD (Ours)96.8299.5692.8896.4235.79

Effectiveness of spectral initialization: Table R3C2 demonstrates that gradient-SVD initialization consistently outperforms random orthonormal initialization, improving suppression by 3.90%, 4.08%, and 7.14% for Male, Young, and Smiling, respectively. This corresponds to a 5.04% improvement in mean suppression, accompanied by a 4.80 dB improvement in PSNR. These findings demonstrate that gradient-SVD provides a more informative attribute-sensitive initialization than arbitrary orthonormal directions, enabling better suppression while preserving higher visual fidelity.

Geometric analysis of learned subspaces: To quantify the influence of subsequent refinement, we measure the principal angles between the initial Gradient-SVD subspace and the final learned subspace. Following Section 3.2 and Appendix A.1, each attribute-specific subspace has rank k = 8 within an ambient image space of dimension d = 49,152. Therefore, comparing the initial and learned subspaces produces eight principal angles, θ₁, θ₂, ..., θ₈, calculated from the singular values of the product of their orthonormal basis matrices. Each principal angle measures alignment along a corresponding pair of principal directions. An angle of 0° indicates perfect alignment, whereas 90° indicates orthogonality. We summarize these eight angles using two complementary metrics:

  1. Mean principal angle: The arithmetic mean of the eight individual angles:

    θ̄ = (1/8) × Σᵢ₌₁⁸ θᵢ

    This indicates their average angular separation.

  2. Subspace overlap: The average squared cosine of the eight principal angles:

    Overlap = (1/8) × Σᵢ₌₁⁸ cos²(θᵢ)

Table R3C2_1: Principal-angle analysis between initial and final learned subspaces.

AttributeOverlap ↑Mean Angle (°)Individual Principal Angles (°)
Male0.41651.716.6, 20.8, 33.7, 49.5, 62.1, 65.7, 80.2, 85.1
Young0.42751.821.5, 27.8, 35.0, 42.1, 58.7, 59.3, 82.4, 87.4
Smiling0.43152.019.9, 22.7, 36.7, 37.3, 52.5, 77.2, 82.7, 86.9

Interpretation of subspace refinement: Table R3C2_1 reveals substantial geometric adaptation, with mean principal angles of approximately 52° across all three attributes. The individual angles exhibit a nonuniform distribution: the two smallest angles remain approximately 17° to 28°, indicating relatively strong alignment along certain principal directions, whereas the two largest reach approximately 80° to 87°, indicating considerable reorientation along others. Thus, the learned subspaces neither simply preserve nor completely discard their initial spectral structure; instead, refinement adapts the gradient-informed representation to the selective privacy objective. This observation is consistent with the original ablation in Table 4(a), where freezing the initial basis reduces male suppression from 96.82% to 32.92%, confirming the importance of learnable refinement.

AuthorsOpenReview ↗

Comparison with Strong Adversarial Attack Methods

3. Comparison with Strong Adversarial Attack Methods

We conducted an additional evaluations using PGD, MI-FGSM, and DI-FGSM, examining white-box attribute suppression, transferability to held-out CNN classifiers, and per-image inference efficiency.

Table R3C3: Comparison with conventional adversarial perturbation methods.

MethodWB Male ↑WB Young ↑WB Smiling ↑WB Mean ↑Held-out CNN Mean ↑Time/Image ↓
PGD81.0098.4087.0088.8011.5110,441.21 ms
MI-FGSM99.8099.80100.0099.8732.009,853.14 ms
DI-FGSM96.8099.8097.8098.1332.719,744.25 ms
GAUSS (Ours)96.8299.5692.8896.4263.839.33 ms

White-box suppression values and held-out CNN flip rates are reported in percentages (%). The held-out CNN mean is averaged over ResNet-50, EfficientNet-B0, and MobileNetV3. Inference time is reported in milliseconds per image for the single-image setting.

Cross-model transferability: The results highlight an important distinction between white-box optimization and cross-model generalization. Conventional adversarial attacks obtain strong mean white-box suppression (88.80% to 99.87%), but their effectiveness decreases considerably on unseen CNN architectures, yielding only 11.51% to 32.71% mean flip rates. In contrast, GAUSS achieves 63.83% mean suppression on held-out CNNs, exceeding the best conventional attack baseline by 31.12%. These findings support the value of evaluating transferability beyond the classifiers directly targeted during perturbation optimization, rather than relying exclusively on white-box suppression performance.

Inference efficiency: GAUSS also demonstrates a substantial computational advantage over conventional iterative attacks. PGD, MI-FGSM, and DI-FGSM require approximately 9.74 to 10.44 seconds per image, whereas our proposed approach GAUSS requires only 9.33 milliseconds in the single-image setting. This corresponds to approximately 1,119×, 1,056×, and 1,044× faster inference than PGD, MI-FGSM, and DI-FGSM, respectively.

We will refine the paper with all these concerns wherever best fit and applicable in the paper.

Reviewer_vASiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GAUSS, a framework for selective facial-attribute suppression using low-dimensional subspaces derived from classifier input gradients. It combines learned orthonormal bases, image-adaptive perturbation coefficients, spatial masks, and language-guided supervision to suppress user-selected attributes while preserving identity and non-target attributes. Experiments report strong classifier suppression and non-target retention, although the privacy interpretation and several experimental inconsistencies require clarification.

Critical strengths
  1. The paper addresses an important trade-off: suppressing selected soft-biometric attributes while preserving identity, non-target attributes, visual quality, and transferability to unseen classifiers.

  2. The combination of shared attribute-specific bases and image-adaptive coefficients provides a structured perturbation mechanism. A single model supports different combinations of user-selected attributes rather than requiring separate models for each request.

  3. GAUSS achieves over 92% white-box suppression for all three evaluated attributes and 98.41% mean non-target retention, outperforming the reported baselines on both measures.

  4. The method reports 35.79 dB PSNR and 0.95 SSIM, alongside smaller identity-verification degradation than PrivAD across the four evaluated verifiers.

  5. Evaluation beyond the white-box setting. Experiments include held-out CNNs, foundation-model probes, post-processing transformations, and cross-dataset tests on LFW and FairFace, providing useful evidence about generalization and limitations.

Critical weaknesses
  1. The white-box metric rewards predictions opposite to the ground-truth binary label. An adversary who knows which attributes are protected could reverse those predictions, potentially recovering the attributes accurately. Thus, high flip rates demonstrate classifier deception, not necessarily reduced attribute recoverability.

  2. Table 4(a)’s 96.82% Male suppression implies 159 failures among 5,000 images, whereas Table 11 counts 653 cases where GAUSS fails and PrivAD succeeds. These cannot describe the same evaluation outcomes, and any differences in checkpoints or protocols need explicit explanation.

  3. The ablations demonstrate the importance of basis refinement and instance adaptation, but do not establish the advantage of spectral initialization over random learnable bases. Matched pixel-space baselines and controlled CNN-only versus text-augmented supervision experiments are also needed to substantiate the claimed mechanisms.

  4. The flip rate is only 26.7% on CLIP-H, below PrivAD’s 28.4%. In the extended robustness evaluation, mean suppression drops from 95.87% without transformation to 58.76% under JPEG Q=90 and 30.73% under combined compression and blur, limiting practical protection.

  5. The reported 1.38M trainable parameters exclude the frozen classifiers whose features are required by the coefficient network. The paper should clarify complete deployment costs and how the attribute class used to orient perturbations is obtained at inference.

Questions for authors
  1. How much attribute information remains recoverable after simple label inversion or score recalibration, and how should the privacy claim be interpreted under such adversaries?

  2. Can you reconcile the suppression rates in Table 4(a) with the paired failure counts in Table 11, specifying the exact checkpoints and evaluation protocols?

  3. Can you provide matched comparisons against random learnable bases and unrestricted pixel-space perturbations, together with an ablation isolating the contribution of language-guided supervision?

  4. Also, address the weakness

4 comments
AuthorsOpenReview ↗

Attribute Prediction Inversion and Privacy Protection

1. Attribute Prediction Inversion and Privacy Protection

We thank the reviewer for highlighting the possibility of recovering protected attributes through prediction inversion. While we acknowledge that reversing predictions may counteract protection under certain conditions, its effectiveness mostly depends on the adversary's knowledge and the consistency of prediction reversal across different models. In a white-box setting, where perturbations are specifically optimized against a known classifier, GAUSS achieves approximately 96% to 98% suppression. Under such conditions, an adversary who knows the protection mechanism and its systematic prediction-reversal behavior could potentially recover the original binary attributes through inversion. However, this assumption does not directly generalize to practical black-box scenarios, where the underlying classifiers, decision boundaries, and responses to protected images may differ substantially from those used during perturbation optimization. Importantly, our cross-model evaluations demonstrate that the prediction flip rates vary considerably across unseen architectures, typically ranging from approximately 50% to 75%. For binary attributes, indiscriminately reversing these predictions would yield only 50% to 75% recovery accuracy, substantially lower than the near-perfect recovery achievable in the white-box setting. Also, when the flip rate approaches 50%, prediction inversion provides little advantage over random guessing. Thus, successful inversion against a known classifier does not establish a universally reliable recovery strategy across heterogeneous inference systems. Also, real-world automated inference pipelines generally lack explicit information about whether a particular image has been protected, which attributes are selected for suppression, and whether the corresponding classifier predictions should be reversed. Applying inversion indiscriminately to both protected and unprotected images may introduce additional prediction errors. For multi-class attributes, the challenge becomes more complex because incorrect predictions do not necessarily correspond to a unique, reversible mapping between the original and predicted classes.

2. Statistical Significance

Tables 1(a) and 4(a) report suppression under individual attribute-protection settings, where Male, Young, and Smiling are protected separately, whereas Table 11 evaluates simultaneous suppression of all three attributes using the joint privacy prompt (p = [1, 1, 1]). Consequently, the 96.82% male suppression rate reported for the individual-attribute setting cannot be directly used to determine the number of failures in the joint-attribute evaluation.

Table R2C2: Corrected paired statistical analysis of GAUSS versus PrivAD under simultaneous three-attribute protection.

Attributen₀₁n₁₀χ²p-valueCohen's h
Male2759290.262.1 × 10⁻²¹0.16
Young1,37471,351.168.6 × 10⁻²⁹⁶0.84
Smiling43225843.384.5 × 10⁻¹¹0.13

The corrected results resolve the previously reported discrepancies. They also confirm that GAUSS significantly outperforms PrivAD across all three attributes when protection is applied simultaneously. We will revise the manuscript to clearly distinguish the two evaluation settings. We will also correct the table captions and statistical results to ensure consistency throughout the paper.

3. Spectral Initialization, and Pixel-Space Baselines We thank the reviewer for suggesting additional controlled comparisons. To isolate the contribution of spectral initialization, we compared gradient-SVD against a randomly initialized orthonormal subspace of identical rank (k = 8), maintaining the same IAP architecture, training objectives, perturbation budget, optimization schedule, and orthogonal refinement procedure.

InitializationMale ↑Young ↑Smiling ↑Mean ↑PSNR (dB) ↑
Random orthonormal92.9295.4885.7491.3830.99
Gradient-SVD (Ours)96.8299.5692.8896.4235.79

Gradient-SVD consistently improves suppression across all attributes, yielding a 5.04% improvement in mean suppression and a 4.80 dB improvement in PSNR. This demonstrates the benefit of attribute-sensitive spectral initialization over arbitrary learnable directions.

We further measured the principal-angle overlap between the initial and final learned subspaces. The resulting overlap values are 0.416, 0.427, and 0.431 for Male, Young, and Smiling, with mean principal angles of approximately 52°. These results indicate substantial task-specific refinement while retaining meaningful alignment with the gradient-informed initialization. This complements the original Table 4(a) ablation, in which freezing the basis reduces Male suppression from 96.82% to 32.92%. We additionally evaluated PGD, MI-FGSM, and DI-FGSM as conventional pixel-space attack baselines. Their measured mean held-out CNN flip rates range from 11.51% to 32.71%, compared with 63.83% reported for GAUSS in Table 3. This provides an additional transferability comparison, although a strictly matched pixel-space ablation must also control perceptual quality, optimization budget, and access to supervision.

4. Robustness

We acknowledge that GAUSS achieves 26.7% suppression on CLIP-H, slightly below PrivAD's 28.4%. However, the extended foundation-model evaluation shows that GAUSS outperforms PrivAD on SigLIP-B (41.0% vs. 28.6%) and achieves 30.13% on DINOv2-L compared with PrivAD's 35.47%. Consequently, GAUSS achieves a slightly higher mean suppression rate across these three evaluators (32.61% vs. 30.82%), while maintaining substantially higher visual fidelity (35.79 dB PSNR vs. 19.40 dB PSNR). These findings highlight the trade-off between cross-model suppression and the preservation of the original facial appearance. Regarding robustness, we acknowledge that mean suppression decreases from 95.87% to 58.76% under JPEG Q=90, and further to 30.73% under combined compression and blurring. This degradation is expected because these operations attenuate the subtle perturbations that underlie attribute suppression, and their combined application causes greater degradation. Nevertheless, GAUSS maintains considerably stronger suppression under additive noise and photometric transformations, as demonstrated in Appendix A.8.

AuthorsOpenReview ↗

Computational Cost

Complete deployment cost: The reported 1.38M trainable parameters correspond only to the GAUSS protection components. At inference, GAUSS additionally requires three frozen ResNet-18 attribute classifiers (33.53M parameters), whose extracted features are used by the coefficient prediction network and whose predictions determine perturbation orientation. Importantly, the additional CNN and CLIP observers used during training are not required at inference. We provide the complete parameter and computational comparison below.

Table R2C5: Complete deployment cost comparison between GAUSS and PrivAD.

MetricGAUSS (Ours)PrivAD
Trainable parameters1.38M97.98M
Frozen classifiers required at inference33.53M—
Total inference-time parameters34.91M97.98M
FP32 weight memory133.2 MiB373.8 MiB
Latency (single attribute)5.1 ms6.5 ms
Latency (all three attributes)9.33 ms19.6 ms

Even after accounting for the frozen classifiers, GAUSS requires approximately 2.8× fewer parameters and lower weight memory than PrivAD. Also, GAUSS achieves approximately 2.1× faster inference under simultaneous three-attribute protection (9.33 ms vs. 19.6 ms), demonstrating its computational efficiency in multi-attribute privacy protection.

Attribute Orientation at Inference: GAUSS does not require ground-truth attribute labels during deployment. Instead, the frozen attribute classifiers predict the attribute classes from the original image, and these predictions determine the perturbation directions in Eq. (5). To evaluate this label-free inference setting, we conducted additional experiments on the same 5,000 test images.

Evaluation protocolMale (%)Young (%)Smiling (%)
Original white-box suppression (Table 1)96.8299.5692.88
Label-free prediction flip rate96.8299.5692.88
Label-free suppression against ground truth96.4896.5688.86

The results demonstrate that GAUSS maintains better suppression without requiring ground-truth attribute annotations, achieving 96.48%, 96.56%, and 88.86% against ground truth for Male, Young, and Smiling, respectively.

Reviewer_vASiOpenReview ↗

Thanks for the response!

Thanks for providing clarifications and experiments, which addressed my concerns. I am happy to increase the score.

Reviewer_tyAWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GAUSS, which creates a low-dimensional subspace for each attribute using SVD of the attribute classifier input gradient and generates bounded perturbations only within it to hide user-selected soft biometric attributes. Training is performed using multiple classifiers and CLIP text direction-based observers, and suppression, non-target retention, identity preservation, transfer, and robustness are evaluated in CelebA-HQ.

Critical strengths
  • The idea of ​​formulating selective privacy as a composition of attribute-specific perturbation subspaces is neat. Thanks to this formulation, a single lightweight model handles arbitrary attribute combinations. It is also commendable that the low-dimensional structure was validated through reproducibility across data partitions without assuming a subspace structure.

  • The analysis that subspaces are nearly orthogonal across architectures, even for the same attribute, is an interesting finding that explains the transfer limitations of surrogate ensembles. This analysis naturally leads to the rationale for introducing text-defined observers.

Critical weaknesses
  • The paper presents application contexts such as authentication, surveillance, and social media, but it lacks specific scenarios regarding who applies protection and who acts as an adversary. Since perturbations do not interfere with humans identifying image attributes, the scope of protection is limited to automated inference. While identity preservation is a key outcome, face recognition templates are known to contain a significant amount of soft biometric information. Nevertheless, the attribute probe on FR embeddings was not evaluated. Furthermore, there is no basis for assuming only oblivious adversaries despite the mechanism being publicly available. Because the objective function reverses predictions rather than de-informing them, an adversary who knows the suppression targets has the potential to invert the output to recover the attributes.

  • The core argument that selectivity comes from subspace constraints is not separated from the effect of retention loss, and there is no bounded-perturbation baseline. In transfer mechanism, there is tension between the authors' analysis and design. If the subspaces for each classifier are nearly orthogonal, a design that confines perturbations to these subspaces may be disadvantageous to transfer, but it is not explained how this is overcome. There are no experiments to support the description attributing this to text observer, and transformer-based classifiers are excluded from the transfer evaluation.

  • The fact that perturbations are concentrated on a low-dimensional, reproducible basis is an advantage in terms of interpretability. However, at the same time, this leaves open the possibility for an adversary to exploit this to detect whether protection is active or to remove it via projection, a point that has not been discussed. It has also not been analyzed whether the direction of perturbations determined by the class contains attribute information.

  • The evaluation of the image transformation itself is extensive. However, even though the suppression effect drops significantly in operations common in the shared pipeline, such as resizing, blurring, and JPEG, the paper merely concludes that it is still effective without discussing what implications it has for actual deployment.

Questions for authors
  • What are the specific deployment scenarios where protection becomes meaningful? In particular, please explain how identity preservation and attribute hiding can coexist within the same FR embedding. Additionally, please tell us how you expect the protection to function against an adaptive adversary who is aware of the proposed mechanism.

  • What is the rationale for distinguishing the contributions of subspace constraints and retention loss, and where can the contribution of text observer be identified? An analysis of how much the learned basis overlaps with the sensitive direction of the held-out classifier is also required.

  • Please explain the transfer mechanism from an attribute classification perspective regarding under what conditions protection is expected to remain effective, including actual pipelines such as commercial attribute inference APIs that are highly likely to be applied in the expected application scenarios.

9 comments
AuthorsOpenReview ↗

Practical Significance and Adversaries

We sincerely thank the reviewer for the careful assessment of our work and for recognizing the novelty of attribute-specific spectral subspaces, the flexibility of their composition, and the motivation behind introducing text-defined observers. We address each concern below.

Practical Significance: The primary motivation behind GAUSS is to protect sensitive facial attributes from unauthorized automated inference while preserving the original visual appearance and social utility of facial images. In real-world scenarios, such as social media sharing, publicly accessible photographs, and online interactions, facial images are increasingly exposed to automated attribute profiling, unauthorized data exploitation, and unwanted surveillance. Our objective is to empower individuals to protect their sensitive attributes against such automated systems without compromising the natural appearance or intended utility of their photographs. Importantly, deceiving human observers is not our objective, as preventing humans from recognizing facial attributes would generally require perceptible modifications, such as facial replacement, substantial manipulation, or visual obfuscation. Such alterations would compromise image authenticity, identity preservation, and social utility, thereby defeating the purpose of our selective protection framework. Instead, GAUSS aims to exploit the difference between human visual perception and automated attribute inference: the protected image should remain visually indistinguishable from, or highly similar to, the original for human observers, while inducing incorrect predictions in automated attribute classifiers. Consequently, individuals can share visually authentic photographs without introducing conspicuous signs of protection, while reducing the effectiveness of the targeted automated inference systems. Therefore, preserving human-perceived facial appearance while disrupting automated attribute inference is an intentional design objective of GAUSS, rather than a limitation of its applicability. Our focus is on achieving this balance between visual fidelity, identity preservation, social utility, and selective protection against automated facial attribute inference.

While we acknowledge that reversing predictions may counteract protection under certain conditions, its effectiveness strongly depends on the adversary's knowledge and the consistency of prediction reversal across different models. In a white-box setting, where perturbations are specifically optimized against a known classifier, GAUSS achieves approximately 96–98% suppression. Under such conditions, an adversary who knows the protection mechanism and its systematic prediction-reversal behavior could potentially recover the original binary attributes through inversion. However, this assumption does not directly generalize to practical black-box scenarios, where the underlying classifiers, decision boundaries, and responses to protected images may differ substantially from those used during perturbation optimization. Importantly, our cross-model evaluations demonstrate that the prediction flip rates vary considerably across unseen architectures, typically ranging from approximately 50–75%. For binary attributes, indiscriminately reversing these predictions would therefore yield only 50–75% recovery accuracy, substantially lower than the near-perfect recovery possible in the white-box setting. Moreover, when the flip rate approaches 50%, prediction inversion provides little advantage over random guessing. Thus, successful inversion against a known classifier does not establish a universally reliable recovery strategy across heterogeneous inference systems. Also, real-world automated inference pipelines generally lack explicit information about whether a particular image has been protected, which attributes are selected for suppression, and whether the corresponding classifier predictions should be reversed. Applying inversion indiscriminately to both protected and unprotected images may introduce additional prediction errors. For multi-class attributes, the challenge becomes more complex because incorrect predictions do not necessarily correspond to a unique, invertible mapping between the original and predicted classes.

AuthorsOpenReview ↗

Subspace Constraints, Retention Loss, and Cross-Model Transferability

We conducted additional experiments to distinguish the contributions of subspace constraints, retention loss, and language-guided supervision.

Subspace constraint versus retention loss: To isolate their individual contributions, we retrained GAUSS without retention loss while keeping all other settings unchanged. Across three seeds and 5,000 test images, mean non-target retention decreased from 98.39 ± 0.02% to 93.71 ± 0.29%, while target suppression remained comparable (96.79%/99.50%/92.94% with retention loss versus 97.67%/99.55%/94.03% without it for Male/Young/Smiling). Also, under matched per-image male optimization without retention loss, the rank-8 subspace achieved 97.98% non-target retention, compared with 90.92% for unrestricted pixel-space optimization. With retention loss, these values reached 98.47% and 99.53%, respectively. These findings demonstrate that the subspace constraint independently promotes attribute selectivity, while the retention objective provides complementary supervision.

Bounded pixel-space comparison: In addition to the PGD, MI-FGSM, and DI-FGSM comparisons reported in our response to Reviewer 1Xhi, we conducted a controlled Male-only experiment comparing unrestricted pixel-space optimization with rank-8 subspace optimization. Both use the same source observers, retention objective, and L-infinity perturbation budget of 8/255 on the [0,1] scale. The per-image optimization baselines use 60 iterations.

Table R1C2_1: Comparison of bounded pixel-space and subspace optimization (Male attribute).

MethodWhite-box (%) ↑Held-out CNNs (%) ↑PSNR (dB) ↑
Pixel-space Adam99.4057.8030.20
Rank-8 subspace Adam92.1276.1335.70
GAUSS (3-seed evaluation)96.7987.3433.14

The results demonstrate that rank-8 subspace optimization improves transferability over unrestricted pixel-space optimization by 18.33% (76.13% vs. 57.80%) and achieves a 5.50 dB higher PSNR. GAUSS further improves held-out CNN suppression to 87.34% in this controlled experiment. This provides evidence that restricting perturbations to informative low-dimensional directions can improve cross-model transferability while preserving image quality.

Transferability despite near-orthogonal subspaces: We agree that the low overlap between classifier-specific gradient subspaces presents a challenge for transferability. Importantly, GAUSS uses the gradient-SVD subspace as an initialization rather than a fixed constraint. The basis is subsequently refined through joint supervision from CNN observers, input diversity, and the CLIP text-defined observer. As demonstrated in our principal-angle analysis, the learned subspaces exhibit an approximately 52° mean angular separation relative to their initialization. Also, the refined bases capture 0.10% to 0.67% of held-out CNN gradient energy, compared with approximately 0.016% for random rank-8 subspaces.

Contribution of language-guided supervision: To isolate the contribution of the text observer, we performed an additional ablation comparing CNN-only supervision against CNN-plus-text supervision, keeping the remaining architecture and training settings identical.

Table R1C2_2: Effect of language-guided supervision on cross-model transfer.

Training observersWhite-box ↑Held-out CNNs ↑DINOv2-L ↑SigLIP-B ↑CLIP-H ↑PSNR (dB) ↑
CNN observers only90.6465.1913.1928.5416.3032.13
CNN + text observer88.4167.0619.3944.0630.5432.12

Including the text observer increases mean suppression across the three held-out foundation models from 19.34% to 31.33%, representing an 11.99% improvement, while maintaining nearly identical PSNR (32.13 vs. 32.12 dB). Although white-box suppression decreases by 2.23%, the substantial improvement on foundation models provides direct empirical evidence that language-guided supervision contributes to cross-architecture generalization beyond CNN-only supervision.

Transformer-based evaluation: We clarify that DINOv2-L, SigLIP-B, and CLIP-H use transformer-based vision encoders, and none of them participate in GAUSS training as held-out evaluators. The text observer uses CLIP ViT-B/16, which is distinct from the CLIP-H evaluator. As reported in our response to Reviewer 1Xhi, the submitted evaluation, extended with DINOv2-L, yields 30.13%, 41.00%, and 26.70% suppression on DINOv2-L, SigLIP-B, and CLIP-H, respectively. While transfer to these models remains weaker than to held-out CNNs, the controlled text-observer ablation demonstrates that semantic supervision substantially improves the transferability of foundation models.

Protection Detectability, Projection-Based Removal, and Attribute Leakage

To investigate whether the learned spectral subspaces can be exploited by an adversary, we conducted additional experiments examining protection detectability, class-conditioned attribute information, and projection-based removal. Protection detectability: We evaluated whether an adversary with access to the learned spectral basis could distinguish protected from clean images. A linear detector operating on basis-projected features achieves 99.69% balanced accuracy (AUROC: 0.9999) on 5,712 test images with identities disjoint from training. This indicates that the low-dimensional structure can reveal the presence of protection to an adversary familiar with the learned basis. Class-conditioned attribute information: In GAUSS, the perturbation direction is determined by the attribute class predicted by the frozen classifiers, as described in Eq. (5), without requiring ground-truth labels during inference. To examine whether this conditioning reveals attribute information, we trained linear probes on the learned subspace projections. The resulting attribute prediction accuracies are 99.35%, 94.31%, and 95.30% for Male, Young, and Smiling on protected images, compared with 90.20%, 74.45%, and 86.09% on clean images, respectively. These findings indicate that class-conditioned perturbations can expose additional attribute information to a basis-aware adversary, despite disrupting the predictions of the original attribute classifiers. Projection-based removal: We further investigated whether projecting images onto the orthogonal complement of the learned subspace could remove the protective perturbations. Although projection reduces the differences between protected and identically projected clean images to 4.68%, 5.22%, and 2.88% for Male, Young, and Smiling, respectively, it also removes naturally occurring attribute-discriminative information. Specifically, classifier accuracy on clean images decreases from 99.66% to 57.38% (Male), 97.00% to 61.10% (Young), and 95.98% to 87.64% (Smiling). Thus, directly removing the learned subspace components does not preserve the original classifier's predictive utility, although more sophisticated adaptive recovery strategies remain possible.

Image Transformation Robustness and Practical Deployment

As demonstrated in Appendix A.8 (Table 10), GAUSS exhibits transformation-dependent robustness, with suppression decreasing under JPEG compression, Gaussian blurring, and resizing.

Transformation-aware robustness: To investigate whether this limitation can be mitigated, we additionally trained GAUSS using Expectation over Transformations (EOT), incorporating randomized JPEG compression (Q=50–95), resizing, and Gaussian blurring during training while keeping all other settings unchanged. We compared standard GAUSS and its EOT-trained variant on 5,000 test images under simultaneous three-attribute protection, measuring prediction changes relative to identically transformed clean images.

Table R1C4: Improvement from EOT training under common image transformations.

TransformationEOT improvement (percentage points) ↑
No transformation+0.27
JPEG Q = 90+0.40
JPEG Q = 70+3.71
JPEG Q = 50+6.24
Gaussian blur+12.35
Resize 64→128+12.47
JPEG Q = 70 + blur+12.87
Resize 32→128+8.10

Deployment implications: EOT improves suppression under Gaussian blurring, resizing, and combined compression-blurring by approximately 12–13%, with only a 0.17 dB reduction in PSNR. However, aggressive processing continues to degrade protection, and EOT reduces unprocessed transfer to held-out CNNs by 5.4% in the same evaluation. These findings highlight a trade-off between transformation robustness and cross-model generalization.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Discussion and Acknowledgement

We sincerely thank the reviewers for providing constructive comments even with these hectic deadline schedules.

But as we are approaching towards the end of this critical discussion period, it would be great if you can revisit our response and provide us positive acknowledgement or provide us the timely opportunity to resolve any new/remaining comments.

We have thoroughly addressed each concern raised by all the reviewers.

AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank all reviewers for their thoughtful and constructive evaluations of our work. We particularly appreciate their recognition of the novelty of attribute-specific spectral subspaces, the flexibility of our selective privacy framework, the effectiveness of our approach in balancing attribute suppression with identity preservation and visual fidelity, and the significance of our cross-architecture analysis. Their valuable feedback has helped us further strengthen the technical and experimental foundations of our proposed approach, GAUSS.

We are especially encouraged that Reviewer vASi explicitly acknowledged that our additional experiments and clarifications addressed their concerns and increased the score to acceptance. We also sincerely appreciate Reviewer tyAW's acknowledgment of our detailed responses and their consideration in the broader discussion among reviewers, as well as Reviewer 1Xhi's recognition of our extensive additional experiments and comparisons, which helped resolve their concern. We are grateful for this constructive engagement and hope our efforts will support a positive overall assessment.

Throughout the rebuttal, we have carefully addressed the concerns raised by each reviewer:

  • Reviewer tyAW: We clarified the real-world applicability of selective facial privacy, conducted experiments that distinguish subspace constraints from retention loss, investigated the transferability mechanism and the text observer, and analyzed basis-aware adversaries and transformation robustness. We also clarified the applicable threat model and its limitations.
  • Reviewer vASi: We addressed prediction inversion, conducted additional initialization and component ablations, extended the foundation model evaluation, and provided a complete analysis of computational and deployment costs. We particularly appreciate the reviewer's positive reassessment following these clarifications.
  • Reviewer 1Xhi: We conducted comprehensive comparisons against conventional adversarial attacks, investigated spectral initialization and subspace refinement, and provided additional evidence of cross-model transferability. We also clarified the distinction between automated attribute suppression and guarantees against fully adaptive attribute recovery.

Collectively, these additional experiments and analyses provide stronger evidence of the novelty, effectiveness, transferability, computational efficiency, and practical significance of our proposed approach, GAUSS, while clarifying its scope of applicability. We sincerely appreciate the reviewers' time and constructive feedback and hope that our comprehensive responses will support a positive final evaluation of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.