acceptodds
Under review as a conference paper at ICLR 2027

InfoAno: Information-Theoretic Face Anonymization

12 views

Abstract

Deep learning-based face recognition (FR) poses a serious threat to individual privacy by exploiting readily available facial data for unauthorized mass surveillance, undermining the public anonymity essential to fundamental civil liberties. Face anonymization serves as a critical countermeasure by removing identity information while preserving useful attributes for legitimate downstream analysis. However, existing approaches struggle with the core privacy–utility conflict: they either lack principled privacy, leaving identity vulnerable to recovery, or lack principled utility, indiscriminately destroying innocuous attributes. In this paper, we pioneer an information-theoretic perspective on face anonymization. We first formulate face anonymization through mutual information and then theoretically expose the identity recovery vulnerability of adversarial-based methods through two failure modes: illusory adversariality and blind optimization. Guided by our theoretical analysis, we propose InfoAno, an information-theoretic face anonymization model that achieves a principled privacy–utility balance. For privacy, InfoAno employs a stochastic substitution mechanism to eliminate illusory adversariality; for utility, it specifies feasible attributes to eliminate blind optimization. We further unify privacy protection and utility preservation within an information bottleneck framework and optimize the resulting objective via deep variational inference with tractable bounds. Extensive experiments validate the superiority of InfoAno, thwarting identity recovery while faithfully preserving expected attributes.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Visual Privacy, Face Anonymization, AI Ethics, Information Theory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NWWfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes InfoAno, an information-theoretic face anonymization framework that formulates face anonymization as minimizing identity leakage while preserving attribute utility, and identifies two failure modes of adversarial-based methods. InfoAno combines stochastic identity substitution, hierarchical attribute partitioning, and an information bottleneck objective optimized through deep variational inference. Experiments show that InfoAno achieves high protection success rates against both arbitrary face recognizers and adaptive recovery attacks, while preserving attributes, face geometry, and head pose more effectively than baseline methods.

Critical strengths

1.Identifies two weaknesses in existing approaches: merely misleading a face recognizer can leave identity information recoverable, and reconstruction losses can preserve identity cues alongside useful attributes. 2.The paper combines stochastic identity substitution with attribute-aware sampling and residual compression. These components address the privacy–utility trade-off and follow naturally from the analysis. 3.The evaluation covers several face recognizers, a commercial API, a recovery attack, and cross-dataset testing. It also measures useful attributes, including facial landmarks and head pose.

Critical weaknesses

1.The evidence for robustness against adaptive identity recovery is limited to a single Pix2Pix reconstructor trained with pixel and identity losses. Stronger evaluation would include other recovery architectures and identity classifiers trained directly on protected images. 2.The main experimental results lack uncertainty estimates and clear repeated-run protocols. The paper leaves variability across training seeds and stochastic identity substitutions unquantified. Reporting repeated-run results, confidence intervals, and the selection protocol would make the comparisons easier to assess. 3.The theory provides limited support for the implemented system. The leakage bound assumes an ideal identity encoder that captures all identity information, but provides no analysis of how encoder errors affect the guarantee. Cosine-based identity exclusion enforces only geometric separation in the chosen embedding space, so statistical dependence may remain. The connection between the theoretical results and the privacy of the released images therefore remains incomplete.

Questions for authors

See weakness

1 comment
AuthorsOpenReview ↗

Gentle Reminder: Rebuttal Feedback and Score Reconsideration

As the rebuttal period is approaching its conclusion, we would be sincerely grateful if you could find time to consider our responses and reconsider your scores in light of the additional evidence and clarifications. We would greatly appreciate any further feedback.

Reviewer_HAyKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates face anonymization as maximizing the attribute information retained in the protected face subject to a bound on identity leakage, both measured by mutual information. It argues that adversarial methods only minimize a variational lower bound of the true identity leakage (“illusory adversariality”) while reconstruction losses retain identity (“blind optimization”). InfoAno decomposes the input into an ArcFace identity embedding and a VIB-regularized residual, replaces the identity with a surrogate sampled under a hard exclusion threshold and soft attribute matching, and restores non-facial regions from the original image. Theorem 1 bounds identity leakage by the VIB rate plus a term that shrinks as the sampling entropy grows. Experimental results demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The paper clearly articulates two failure modes of adversarial anonymization, explaining why such methods remain vulnerable to identity recovery.
  2. The stochastic substitution mechanism with a hard exclusion threshold is simple and well justified.
Critical weaknesses
  1. The theory does not cover the final published output. Theorem 1 holds for the pre-refinement face Y under an ideal-encoder assumption, whereas the released Y_r retains hair and background copied directly from the original input X. Remark 5 acknowledges that the Markov chain need not hold for Y_r and does not claim the bound for it, and Section 5.2 attributes InfoAno’s lower PSR on IRSE50 relative to k-anonymity methods to these retained regions. The claim of “principled privacy” therefore still rests on empirical evaluation, the very approach the paper critiques in adversarial methods.
  2. The compared baselines are somewhat weak. The generative anonymization methods the paper itself cites (DeepPrivacy, CIAGAN, RiDDLE, DiffPrivate) are not compared. Moreover, InfoAno underperforms StyleAdv on shape disparity, pitch, and yaw (Tables 1 and 2).
  3. The adaptive attacker is limited to a single Pix2Pix reconstructor. A more direct attack, training a recognizer on protected faces labeled with their original identities, is not evaluated.
  4. Surrogate identities may be drawn from the training identity database (Appendix D.3.1), so a protected face may carry the identity of a real individual from CelebA.
Questions for authors
  1. Please report the PSR of InfoAno and StyleAdv against a recognizer fine-tuned on pairs of Y_r and the original identity.
  2. Please include at least one generative anonymization method in Figure 8 and Tables 1 and 2.
  3. What is the PSR of Y prior to refinement?
  4. Are surrogate identities drawn from the training identity database or from the virtual identity database?
  5. How is tau in Eq. 14 selected, and how does the PSR vary with tau?
Reviewer_oLTmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents InfoAno, an information-theoretic framework for face anonymization that aims to suppress identity leakage while preserving useful facial attributes. The authors formulate the privacy–utility trade-off using mutual information and identify two limitations of adversarial anonymization: illusory adversariality and blind optimization. InfoAno combines stochastic identity substitution, hierarchical attribute partitioning, and a variational information bottleneck objective. Experiments on public datasets are used to show the performance of InfoAno.

Critical strengths
  1. The paper explore the problem of face anonymization from the viewpoint of information-theoretic which are rarely explored previously.

  2. InfoAno combines stochastic identity substitution with attribute-aware preservation to explicitly address the privacy–utility trade-off.

Critical weaknesses
  1. Assumption 1 in Section 3.1 treats the identity encoder as ideal, fully capturing identity while separating it from non-identity factors. This strong assumption may not hold for practical ArcFace embeddings.

  2. Theorem 1 in Section 4 applies to the pre-refinement image, whereas the Image Refinement step in Section 3.3 restores regions from the original image. Maybe, the privacy bound therefore does not cover the complete released image.

  3. In Section 1&2 and Fig.1, only limited anonymization methods are discussed, especially the related/recent ones. Discussing them would be helpful, such as: [a]Face De-Identification: State-of-the-Art Methods and Comparative Studies, IEEE Transactions on Broadcasting, 26. [b]Face Anonymization Made Simple, WACV, 25 [c]Facial Identity Anonymization via Intrinsic and Extrinsic Attention Distraction, CVPR, 24

  4. The adaptive recovery evaluation in Section 5.2 and Figure 8(b) primarily uses a Pix2Pix-based attacker. Stronger feature-distillation are not investigated, e.g. diffusion-based or identity-linkage attacks.

  5. The Baselines paragraph in Section 5.1 mainly compares k-anonymity, differential privacy, and adversarial perturbation methods. Maybe, comparisons with generative approaches are expected.

  6. The ablation study in Section 5.4 and Appendix F focuses on ID-UpBlock and image refinement. How about the effects of stochastic substitution, attribute-aware sampling, and the information bottleneck objective.

Questions for authors

See weaknesses above.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Contributions

We sincerely thank all reviewers and the Area Chair for their thoughtful evaluations and constructive feedback. As the rebuttal period approaches its conclusion, we would like to summarize the key contributions of InfoAno and the additional efforts made to address the reviewers' concerns.

Contributions of InfoAno

1. Novelty. Our core novelty is to revisit face anonymization from an information-theoretic privacy perspective, encompassing problem formalization, vulnerability analysis, model design, and practical implementation.

  • Theoretical Insights: We formulate the privacy–utility trade-off using mutual information and identify two fundamental failure modes of adversarial anonymization.
  • Methodological Connection: We translate these theoretical insights into InfoAno's design and validate the resulting framework through extensive experiments, establishing a coherent connection between theoretical analysis, vulnerability discovery, model design, and practical implementation.
  • Supporting Evidence: As Reviewer oLTm noted, this perspective has been "rarely explored previously." Reviewer HAyK also recognized that our work "clearly articulates two failure modes of adversarial anonymization," while Reviewer NWWf highlighted the corresponding methodological contributions. These observations underscore the novelty of both our theoretical insights and framework design.

2. Presentation. We have carefully organized the manuscript to establish a clear connection between theoretical motivation, methodological development, and experimental validation, supported by illustrative figures and detailed explanations. We appreciate that all three reviewers rated the presentation as "Clear and easy to follow." Following their suggestions, we have further improved the clarity of our theoretical assumptions, experimental protocols, and related-work discussions in the revised manuscript.

3. Soundness. Our work combines principled theoretical analysis with extensive empirical validation, supported by three aspects:

  • Original Evidence: Our original evaluation covers multiple face recognizers, adaptive recovery attacks, cross-dataset generalization, and attribute preservation. Reviewer HAyK acknowledged in the paper summary that "Experimental results demonstrate the effectiveness of the proposed method." Reviewer oLTm noted that InfoAno can "explicitly address the privacy–utility trade-off." Similarly, Reviewer NWWf recognized that our proposed components "address the privacy–utility trade-off and follow naturally from the analysis."

  • Theoretical Clarification: We have clarified the assumptions and applicability of our theoretical guarantees, emphasizing their role in guiding practical implementation rather than claiming unconditional privacy. We further extended our analysis to account for non-ideal identity encoders and explicitly characterize how potential contextual leakage from image refinement affects the theoretical bound.

  • Strengthened Experimental Evidence: Following the reviewers' suggestions, we additionally evaluated DeepPrivacy2, introduced an ArcFace-based adaptive embedding regression attack, conducted ablations of three core components, and reported repeated-run evaluations with confidence intervals.

4. Significance. We believe InfoAno offers meaningful contributions to the broader ICLR community in three aspects:

  • Practical Impact: Mitigating the risks of unauthorized facial recognition, mass surveillance, and persistent identity tracking, thereby helping safeguard individuals' privacy, anonymity, and fundamental freedoms in the digital age.

  • Research Insights: Revealing fundamental vulnerabilities of adversarial anonymization and providing new theoretical insights into identity leakage and privacy–utility trade-offs.

  • Broader Value: Introducing an information-theoretic perspective that may inspire future research on privacy-preserving representation learning, responsible data sharing, and trustworthy AI.

We emphasize that InfoAno's contribution extends beyond individual benchmark improvements. Its central value lies in establishing a coherent connection between information-theoretic problem formulation, fundamental vulnerability analysis, principled model design, and extensive empirical validation, providing a systematic approach to understanding and addressing the privacy–utility trade-off in face anonymization. We believe this offers novel and meaningful insights to the ICLR community, consistent with the conference's emphasis on both scientific rigor and broader research significance.


AuthorsOpenReview ↗

Summary of Rebuttal

We sincerely thank all reviewers and the Area Chair for their thoughtful evaluations and constructive feedback. As the rebuttal period approaches its conclusion, we would like to summarize the key contributions of InfoAno and the additional efforts made to address the reviewers' concerns.

Key Concerns Addressed During Rebuttal

We have carefully reviewed and summarized the four major concerns raised by the reviewers. Through additional experiments, theoretical extensions, and detailed clarifications, we believe these concerns have been substantially addressed.

1. Generative Anonymization Baseline. Following the reviewers' suggestions, we additionally evaluated DeepPrivacy2 under the same protocol. While DeepPrivacy2 achieves higher privacy protection (96.86% vs. 91.47% PSR), InfoAno demonstrates better attribute preservation (90.32% vs. 86.97% AUC), facial geometry, and head-pose consistency. These results further substantiate InfoAno's effectiveness in balancing privacy and utility and have been incorporated into the revised manuscript's appendix due to space constraints.

2. Adaptive Identity Recovery Attacks. We clarified that our recovery-based threat model was explicitly defined in the original submission (Fig. 10 and Appendix B.3), in which the adversary trains an adaptive image reconstructor using paired original--protected samples (X,Y)(X,Y) to reconstruct original facial appearances and subsequently recover identity features using the original face recognizer. We further strengthened the experimental evidence in two aspects:

  • Original Evaluation: Our Pix2Pix-based recovery attack directly evaluates the predefined threat of reconstructing original identity features through face recovery. Its effectiveness is supported by the substantial degradation of DP-based and adversarial baselines under the same attack, whereas InfoAno maintains strong protection.
  • Enhanced Evaluation: Following the reviewers' suggestions, we additionally trained an ArcFace-based adaptive recognizer using embedding regression directly on protected images. InfoAno maintains 83.4% PSR, compared with only 5.6% for StyleAdv, providing complementary evidence of robustness.
  • Scope and Limitations: We acknowledge that additional recovery architectures, such as diffusion-based models, could further strengthen our empirical evidence. However, these experiments were difficult to implement within the limited rebuttal period. Direct identity classification represents a distinct attack setting beyond our predefined recovery-based threat model. We consider these evaluations valuable extensions rather than prerequisites for validating our method under the threat model explicitly defined in our original submission.

3. Ideal Identity-Encoder Assumption. We clarified that the ideal identity encoder is introduced for theoretical analysis and principled guidance, rather than claiming unconditional privacy guarantees for practical implementations. Following the reviewers' suggestions, we further extended Theorem 1 by incorporating encoder uncertainty ϵE=H(S∣ZS)\epsilon_E=H(S\mid Z_S), explicitly characterizing its influence on the privacy bound.

Importantly, our theoretical analysis is intended to guide practical model design, while empirical evaluations validate the resulting implementation under non-ideal conditions.

4. Theoretical Coverage of the Final Released Output. Our original analysis focuses on the pre-refinement output YY. Our threat model focuses on facial identity recognition based on facial features, while acknowledging that restored non-facial regions, such as hair and background, may contain contextual identity information.

Following the reviewers' suggestions, we introduced an additional identity-leakage term I(C;S∣Y)I(C;S\mid Y) to explicitly quantify potential identity leakage from restored non-facial information. We further specified the additional condition under which the original privacy bound extends to the final refined output YrY^r. Importantly, this additional condition does not follow from the ideal identity-encoder assumption alone. Empirically, refinement reduces average PSR by only 1.30 percentage points, providing complementary evidence of its limited impact on the evaluated privacy protection.


We have incorporated these theoretical clarifications and threat-model refinements into the updated PDF, including the extended analysis of the final refined output (Appendix, Remark 5) and the clarified recovery-based threat model (Fig. 10 and Appendix B.3).

We sincerely appreciate the reviewers' time, effort, and constructive feedback throughout the rebuttal process. We respectfully hope that these clarifications, additional results, and the revised manuscript will be considered in the final assessment of InfoAno.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.