acceptodds
Under review as a conference paper at ICLR 2027

InvFace: Privacy-Preserving Face Recognition Using Identity-Distilling Inversion

Abstract

Sharing unprotected images for face recognition can expose sensitive visual information beyond what is needed for identity comparison. Existing protection methods transform face images into identity-preserving and visual-obscuring representations, yet their effectiveness is validated mainly empirically, leaving privacy disclosure unquantified. This paper proposes a stepwise paradigm that decomposes monolithic protection into explicit, atomic transitions whose cumulative effects are auditable. It is instantiated as a novel protection method, InvFace. InvFace performs identity-distilling inversion, where an identity-conditioned diffusion model, inverted without the identity condition, produces noise-like yet recognition-exploitable latents. This is followed by mirrored classifier-free guidance, which strengthens identity retention and disrupts direct recovery, and quantization, which makes the representation finite-valued. Taken together, this design admits information-theoretic bounds on visual disclosure beyond identity and generic reconstruction error. Extensive experiments show that InvFace matches leading prior works in recognition utility while providing stronger resistance to multiple reconstruction attacks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
face recognition, privacy-preserving face recognition, diffusion model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DrEROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes InvFace, a privacy-preserving face recognition method based on identity-distilling diffusion inversion. Unlike conventional single-step representation transformations, the authors introduce a stepwise privacy-preserving paradigm consisting of multiple explicit transformation steps. By leveraging the inversion process of an identity-conditioned diffusion model, the proposed method generates protected representations that preserve identity recognition capabilities while concealing visual information.

Critical strengths

The paper explores the use of identity-conditioned diffusion inversion for privacy-preserving face recognition. By combining identity-distilling inversion, mirrored classifier-free guidance (mCFG), and quantization, the authors aim to achieve a balance between identity recognition performance and visual information concealment.

Critical weaknesses
  1. The paper attributes its main contribution to privacy quantification to sequential small-step transitions, arguing that explicit stepwise transformations enable the analysis and accumulation of information changes across individual stages, thereby providing a computable theoretical upper bound on residual privacy leakage. However, if the same form of information upper bound can be obtained for an arbitrary monolithic protected representation under the same quantization setting by directly estimating its terminal within-identity variance, the authors need to further clarify what additional theoretical properties, analytical capabilities, or tighter privacy bounds the proposed stepwise paradigm offers for privacy quantification.

  2. The paper employs mCFG to alter the diffusion inversion trajectory, preventing the protected representation from being directly recovered through the original unconditional or conditional sampling branches. However, the inability of the original sampling branches to directly reverse the mCFG trajectory does not necessarily imply that a recovery model specifically trained or optimized to reverse mCFG would also fail.

Questions for authors
  1. The within-identity variance statistic used in Theorem 2 ultimately equals the terminal within-identity variance (Vr), since the variance and covariance recursions across the transformation steps telescope exactly to this terminal statistic. Could the same form of information-theoretic upper bound therefore be obtained for an arbitrary monolithic protected representation under the same quantization setting simply by directly estimating its terminal within-identity variance? If so, could the authors further clarify what essential theoretical advantages the proposed stepwise paradigm provides for privacy quantification over directly analyzing the terminal representation, beyond merely decomposing the same terminal statistic into multiple steps?

  2. Table 4 reports an information leakage upper bound that is close to the maximum information capacity of the quantized representation, while Table 5 reports a semantic reconstruction error lower bound that is nearly zero. Could the authors clarify how these results can be interpreted as meaningful privacy guarantees?

  3. Theorems 2 and 3 provide an upper bound on the conditional mutual information of the protected representation and a corresponding lower bound on reconstruction error, respectively. However, whether these theoretical results provide meaningful quantitative guarantees regarding practical privacy risks remains to be further validated. Specifically, Table 4 reports an information upper bound of approximately 7.37 bits per representation coordinate, whereas the protected representation uses 8-bit quantization. Since this upper bound is relatively close to the maximum information capacity of the quantized representation, it places only a relatively loose constraint on residual information leakage. Therefore, although the proposed theoretical framework provides computable information-theoretic expressions, the paper has not sufficiently demonstrated that these quantitative results can effectively characterize differences in practical privacy risks across different protection configurations and PPFR methods.

4 comments
AuthorsOpenReview ↗

Response to Reviewer DrER (1/2)

We appreciate the reviewer’s helpful comments and thank them for highlighting InvFace’s approach to balancing recognition utility and visual privacy. Below, we provide additional results and clarifications to address the reviewer’s concerns.

W1/Q1, Stepwise analysis of information upper bound: We agree that the information upper bound in the original submission can be obtained monolithically and that the additional analytical capabilities need further justification. We have revised Thm. 2 to explicitly incorporate stepwise analysis, which also gives us an analytical basis for choosing the endpoint TT.

(W1/Q1-1) Stepwise derivation of Thm. 2. We here provide a proof sketch; please refer to Appendix B.3 of our revised paper for the full derivation. In the original derivation, summing the stepwise variance changes recovers the same variance that can be measured directly at the endpoint. In the revision, we consider two images of the same person. Their latent states and identity conditions can both differ. For the t-th step FtF_t, we first change the state while holding the condition fixed, then change the condition while holding the new state fixed:

δtst=Ft(z,C)−Ft(z′,C),δtcond=Ft(z′,C)−Ft(z′,C′).\delta_t^{\rm st}=F_t(z,C)-F_t(z',C),\qquad\delta_t^{\rm cond}=F_t(z',C)-F_t(z',C').

Their sum is the actual difference between the two outputs. Squaring and averaging this difference gives a state term, a condition term, and a signed cross term. Using the RMS state sensitivity ρtst\rho_t^{\rm st}, the condition contribution χt\chi_t, and the positive part of the normalized cross term ηtsc\eta_t^{\rm sc}, we propagate the variance budget as

Bt=(ρtst)2Bt−1+χt2+2[ηtsc]+ρtstBt−1,χt,B0=V0.\mathcal{B}_t = (\rho_t^{\mathrm{st}})^2 \mathcal{B}_{t-1} + \chi_t^2 + 2[\eta_t^{\mathrm{sc}}]_+ \rho_t^{\mathrm{st}} \sqrt{\mathcal{B}_{t-1}}\\,\chi_t, \qquad \mathcal{B}_0 = V_0.

Taking the positive part makes the update nondecreasing in the preceding variance, so the recursion preserves the upper bound at every step. We then use the same quantization and entropy argument to obtain UΔU_\Delta from BT\mathcal B_T. The stepwise analysis adds a diagnostic capability: it shows where changes in within-identity variance occur and how state propagation, condition variation, and their interaction contribute to those changes.

Based on the new derivation, using real samples from CASIA-WebFace, we estimate the revised upper bound at 7.23 bits/coordinate, which is comparable to the original estimate.

(W1/Q1-2) Choice of the inversion endpoint. The revised derivation of Thm. 2 also helps us choose the endpoint TT. As inversion progresses, the latents become more noise-like and image-specific facial details become harder to discern. Intuitively, one might expect further inversion steps (increasing TT) to always tighten the information upper bound. However, both original and revised numerical estimates reveal a sweet point, where the bound reaches its minimum at around T=500T=500 and rises at T=550T=550.

TU^Δ/(nln⁡2)\widehat{U}_{\Delta}/(n\ln 2)(bits/coord.)Condition & cross termState contraction term
507.26200.0090-0.0159
2507.24650.0047-0.0145
4507.23140.0143-0.0148
5007.22990.0137-0.0151
5507.24490.0284-0.0133
7507.24900.0138-0.0138
10007.26230.0147-0.0136

The stepwise recursion helps explain this reversal. With the mCFG weight fixed, each update propagates the differences already present in the latent state, introduces differences due to the identity condition, and allows the two to interact. Subtracting the preceding variance budget from both sides of the recursion gives

\underbrace{\big[(\rho_t^{\mathrm{st}})^2 - 1\big]\mathcal{B}_{t-1}}_{\text{state contraction term}} + \underbrace{\chi_t^2 + 2[\eta_t^{\mathrm{sc}}]_+ \rho_t^{\mathrm{st}} \sqrt{\mathcal{B}_{t-1}}\\,\chi_t}_{\text{condition and cross terms}}.$$ Even if an update (increase of $T$) continues to contract the state, a similar contraction coefficient removes less variance once the preceding variance budget is smaller. The condition and cross terms can then outweigh that reduction. This is what we observe around the turning point. At $T=500$, state contraction outweighs the condition and cross terms (0.0137 vs. 0.0151 by magnitude). At $T=550$, the condition and cross terms are larger and outweigh state contraction (0.0284 vs. 0.0133). This explains the increase in the propagated variance budget at that update and supports choosing $T=500$, where the estimated bound is lowest.
AuthorsOpenReview ↗

Response to Reviewer DrER (2/2)

W2, trained model for trajectory reversal: We study a recovery model specifically trained to reverse mCFG.

(W2-1) Trajectory reversal with DM. We consider a white-box attacker with full knowledge of the step-wise protection pipeline and access to both the protected latent and the exact identity embedding CC. The attacker trains a diffusion model on pairs of protected latents and their corresponding original images, using the protected latent as the initial noisy state and the original image as the generation target. At each recovery step, the model takes the same current latent and computes two updates, one without CC and one conditioned on CC, and combines the two updated latents using mCFG. The attack result is shown in Fig. 7(a)(ii) of our revised main text. As OpenReview responses support only plain text, please refer to the revision for visualizations. Even with targeted training and access to the exact identity embedding, the attacker DM failed to faithfully reconstruct the original faces.

(W2-2) Other types of attacks. We have also supplemented our evaluation with other types of attacks to more comprehensively assess the protection offered by InvFace, including two optimization-based attacks and direct trajectory reversal given the exact CC. InvFace is resilient to all the attacks studied. If the reviewer is interested in these analyses, please refer to our responses to Reviewer eJjC’s (Q1-1), (Q1-2), and (Q5-2).

Q2/Q3, Informativeness of numerical estimates: We provide a clearer interpretation of the numerical estimates of bounds below.

(Q2/Q3-1) The information upper bound. We acknowledge that the estimate of 7.23 bits/coordinate is close to the 8-bit capacity ceiling. That said, we note that this does not imply a privacy deficiency and that the bound can be further tightened. From a privacy perspective, since quantization itself is a protection step, the estimate should be read as showing how much inversion further tightens the bound, with the quantized latent serving as the reference point instead of an unprotected baseline.

The bound is also conservative in its current form. One source of looseness is that it uses pixel-wise variance without accounting for correlations between pixels. For example, a single change in facial pose can affect many pixels at once. Each of these changes contributes to the variance, even when they reflect the same underlying information. The actual residual information can therefore be much lower than the bound suggests, which is consistent with the strong resistance to reconstruction we actually observe in our experiments. We believe the bound could be tightened by explicitly modeling these correlations or by accounting for exactly which attack-relevant information is retained. However, both refinements would require additional assumptions in Thm. 2 and reduce its generality. We hence leave them to future work.

(Q2/Q3-2) The semantic reconstruction lower bound. The underlying reason of its nearly zero estimate is, Thm. 3 allows for the worst case for privacy, in which the attacker receives exactly the information most useful for reconstructing the target face. For example, a bit that helps the attacker determine the subject’s skin color is far more useful than a bit about the background. In this example, the bound gives the attacker the bit about skin color. In practice, some of the information the attacker obtains does little to help reconstruction. This is consistent with our findings in Sec. 4.5, where the semantic attack on InvFace is unsuccessful even though the estimated lower bound is low.

(Q2/Q3-3) The pixel reconstruction lower bound. Its numerical estimate is of practical significance. The estimate was 0.23 in our original submission, which corresponds to a mean absolute error of 11.5% of the [−1,1][-1,1] pixel value range. After accounting for pixel correlations in the revision (as suggested by Reviewer eJjC), we obtain a revised estimate of 0.156. As with semantic reconstruction, the bound considers the worst case for privacy. In practice, the attack evaluated in Sec. 4.4 yields poor reconstructions.

(Q2/Q3-4) Numerical estimates for configuration choice and SOTA comparison. The estimated information upper bound provides an analytical basis for choosing protection configurations. As shown in W1/Q1, the stepwise analysis explains how the bound varies across inversion endpoints and guides the choice of TT. Since the evaluated PPFR SOTAs do not report comparable numerical privacy bounds, we compare their protection experimentally through the pixel and semantic reconstruction experiments in Secs. 4.4 and 4.5.

Thank you for the detailed rebuttal and the additional experiments targeting mCFG. I appreciate these efforts, which further strengthen the paper's experimental evaluation. In addition, the authors have provided further theoretical derivations in the revised manuscript, improving the interpretability of the stepwise analysis. Overall, the rebuttal has addressed most of my previous concerns.

AuthorsOpenReview ↗

Thank You for Your Follow-Up

Thank you for your thoughtful and positive follow-up. Your acknowledgment of our efforts means a lot to us. We are glad that our rebuttal has addressed most of your concerns. In light of these improvements, would you be willing to consider raising your rating?

Many thanks!

Reviewer_E65nOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes InvFace, a privacy-preserving face recognition (PPFR) framework that converts face images into visually obscured but identity-preserving representations. It combines identity-distilling diffusion inversion, mirrored classifier-free guidance (mCFG), and quantization.

Identity-distilling inversion produces noise-like latents suitable for face recognition. mCFG improves identity retention while disrupting direct reversal of the inversion trajectory. Quantization enables information-theoretic bounds on residual disclosure beyond identity and on reconstruction error.

Experiments compare InvFace with existing PPFR methods on standard recognition benchmarks and evaluate its resistance to feature and semantic reconstruction attacks.

Critical strengths
  1. Meaningful methodological contributions with a clear motivation.

Using identity-distilling diffusion inversion to produce protected representations is an interesting alternative to conventional feature pruning and learned obfuscation.

The mCFG mechanism addresses a concrete vulnerability of diffusion inversion: the original sampling trajectory may recover the input image. The proposed modification breaks the correspondence with the native conditional and unconditional sampling trajectories while improving recognition utility.

The information-theoretic formulation is also valuable. Rather than evaluating privacy exclusively through visual inspection or specific attacks, the authors attempt to bound the residual information disclosed beyond identity.

  1. Strong experimental results and comprehensive comparisons.

The paper evaluates recognition utility on seven benchmarks and compares InvFace with 12 PPFR methods. Its average recognition accuracy drop is approximately 1% relative to the unprotected baseline, making it competitive with leading approaches.

More importantly, InvFace demonstrates stronger resistance to the evaluated reconstruction attacks. It achieves the lowest reconstruction SSIM (0.37) among the compared methods and the lowest semantic reconstruction similarity (0.22).

The inversion-strategy comparisons and ablations of diffusion endpoints, guidance scales, and quantization also support the effectiveness of the proposed components.

Overall, these results demonstrate a favorable empirical privacy–utility trade-off.

Critical weaknesses

The practical benefit of stepwise privacy quantification is not sufficiently demonstrated.

The paper emphasizes stepwise privacy auditing as a central contribution. Section 3.5 derives an information upper bound using the accumulated within-identity variance of the inversion trajectory.

Specifically, the transition statistics satisfy

V_t = V_(t−1) + v_t + 2c_t,

and the final variance is used to bound

I(X; R̂ | S) ≤ UΔ.

However, changes in within-identity variance do not directly measure changes in residual mutual information. The final information bound can also be obtained from the statistics of the terminal representation without explicitly decomposing the intermediate transitions.

Therefore, the additional value of the stepwise decomposition remains unclear. In particular, the paper does not quantitatively demonstrate how intermediate transitions can guide the choice of an inversion endpoint or a desired privacy–utility trade-off.

The numerical bounds also deserve clearer interpretation. Appendix B reports an information upper bound of 7.37 bits per coordinate for an 8-bit representation, together with a positive pixel-reconstruction error lower bound but a nearly zero semantic-reconstruction error lower bound. These results establish computable bounds, but their practical informativeness varies considerably.

This does not undermine the strong empirical reconstruction resistance. However, more direct evidence is needed to substantiate the claimed practical value of stepwise privacy quantification.

Questions for authors
  1. Can the proposed framework quantify privacy at intermediate inversion endpoints?

For example, could the authors compute UΔ(t) for protected representations obtained by stopping and quantizing at different endpoints, such as T = 250, 500, and 1000?

Comparing these bounds with recognition accuracy and reconstruction error would directly test whether stepwise auditing can guide endpoint selection.

  1. What advantage does transition-wise variance accumulation provide over directly measuring the final representation?

Since the final information bound depends on the terminal within-identity variance, could the authors clarify what additional privacy information is obtained from the intermediate transitions?

It would also be helpful to summarize the key numerical bounds from Appendix B in the main paper and explain their practical implications.

  1. How sensitive is mCFG’s reconstruction resistance to the accuracy of the identity condition?

Appendix C.1 evaluates adaptive reversal using a condition estimate close to the true condition. Could the authors report reconstruction quality across different condition-estimation errors, including an exact-condition oracle setting?

This would help distinguish protection due to trajectory mismatch from protection that relies on uncertainty about the sample-specific condition.

Additional Comments

The paper could more explicitly distinguish visual obscurity, resistance to evaluated reconstruction attacks, and information-theoretic privacy guarantees. These are complementary but different properties.

The main paper would also benefit from including the most important numerical privacy-bound results, rather than leaving their interpretation primarily to the appendix.

Overall Assessment

InvFace presents an interesting and well-motivated approach to privacy-preserving face recognition. The methodological design is meaningful, and the experiments demonstrate competitive recognition utility with strong resistance to the evaluated reconstruction attacks.

The primary limitation is that the practical advantage of stepwise privacy quantification is not yet sufficiently established. Nevertheless, the proposed framework and empirical results represent valuable contributions.

4 comments
AuthorsOpenReview ↗

Response to Reviewer E65n (1/2)

We appreciate the reviewer’s constructive feedback and positive assessment of InvFace’s methodological contributions and experimental results. We address the remaining concerns below.

W1/Q1, Stepwise analysis of information upper bound: The reviewer suggests that the stepwise analysis was not sufficiently demonstrated in the initial submission. We indeed agree. We have revised Thm. 2 to incorporate stepwise analysis explicitly, which also gives us a new insight to guide the choice of endpoint TT.

(W1/Q1-1) Stepwise derivation of Thm. 2. We here provide a proof sketch; please refer to Appendix B.3 of our revised paper for the full derivation. In the original derivation, summing the stepwise variance changes recovers the same variance that can be measured directly at the endpoint. In the revision, we consider two images of the same person. Their latent states and identity conditions can both differ. For the t-th step FtF_t, we first change the state while holding the condition fixed, then change the condition while holding the new state fixed:

δtst=Ft(z,C)−Ft(z′,C),δtcond=Ft(z′,C)−Ft(z′,C′).\delta_t^{\rm st}=F_t(z,C)-F_t(z',C),\qquad\delta_t^{\rm cond}=F_t(z',C)-F_t(z',C').

Their sum is the actual difference between the two outputs. Squaring and averaging this difference gives a state term, a condition term, and a signed cross term. Using the RMS state sensitivity ρtst\rho_t^{\rm st}, the condition contribution χt\chi_t, and the positive part of the normalized cross term ηtsc\eta_t^{\rm sc}, we propagate the variance budget as

Bt=(ρtst)2Bt−1+χt2+2[ηtsc]+ρtstBt−1,χt,B0=V0.\mathcal{B}_t = (\rho_t^{\mathrm{st}})^2\mathcal{B}_{t-1} + \chi_t^2 + 2[\eta_t^{\mathrm{sc}}]_+ \rho_t^{\mathrm{st}}\sqrt{\mathcal{B}_{t-1}}\\,\chi_t, \qquad \mathcal{B}_0=V_0.

Taking the positive part makes the update nondecreasing in the preceding variance, so the recursion preserves the upper bound at every step. We then use the same quantization and entropy argument to obtain UΔU_\Delta from BT\mathcal B_T. The stepwise analysis adds a diagnostic capability: it shows where changes in within-identity variance occur and how state propagation, condition variation, and their interaction contribute to those changes.

Based on the new derivation, using real samples from CASIA-WebFace, we estimate the revised upper bound at 7.23 bits/coordinate, which is comparable to the original estimate.

(W1/Q1-2) Choice of the inversion endpoint. The revised derivation of Thm. 2 also helps us choose the endpoint TT. As inversion progresses, the latents become more noise-like and image-specific facial details become harder to discern. Intuitively, one might expect further inversion steps (increasing TT) to always tighten the information upper bound. However, both original and revised numerical estimates reveal a sweet point, where the bound reaches its minimum at around T=500T=500 and rises at T=550T=550.

TTU^Δ/(nln⁡2)\widehat{U}_{\Delta}/(n\ln 2) (bits/coordinate)Condition & cross termState contraction term
507.26200.0090-0.0159
2507.24650.0047-0.0145
4507.23140.0143-0.0148
5007.22990.0137-0.0151
5507.24490.0284-0.0133
7507.24900.0138-0.0138
10007.26230.0147-0.0136

The stepwise recursion helps explain this reversal. With the mCFG weight fixed, each update propagates the differences already present in the latent state, introduces differences due to the identity condition, and allows the two to interact. Subtracting the preceding variance budget from both sides of the recursion gives

\underbrace{\big[(\rho_t^{\mathrm{st}})^2 - 1\big]\mathcal{B}_{t-1}}_{\text{state contraction term}} + \underbrace{\chi_t^2 + 2[\eta_t^{\mathrm{sc}}]_+ \rho_t^{\mathrm{st}} \sqrt{\mathcal{B}_{t-1}}\\,\chi_t}_{\text{condition and cross terms}}.$$ Even if an update (increase of $T$) continues to contract the state, a similar contraction coefficient removes less variance once the preceding variance budget is smaller. The condition and cross terms can then outweigh that reduction. This is what we observe around the turning point. At $T=500$, state contraction outweighs the condition and cross terms (0.0137 vs. 0.0151 by magnitude). At $T=550$, the condition and cross terms are larger and outweigh state contraction (0.0284 vs. 0.0133). This explains the increase in the propagated variance budget at that update and supports choosing $T=500$, where the estimated bound is lowest.
AuthorsOpenReview ↗

Response to Reviewer E65n (2/2)

W2/Q2, Informativeness of numerical estimates: We provide a clearer interpretation of the numerical estimates of bounds below.

(W2/Q2-1) The information upper bound. We acknowledge that the estimate of 7.23 bits/coordinate is close to the 8-bit capacity ceiling. That said, we note that this does not imply a privacy deficiency and that the bound can be further tightened. From a privacy perspective, since quantization itself is a protection step, the estimate should be read as showing how much inversion further tightens the bound, with the quantized latent serving as the reference point instead of an unprotected baseline.

The bound is also conservative in its current form. One source of looseness is that it uses pixel-wise variance without accounting for correlations between pixels. For example, a single change in facial pose can affect many pixels at once. Each of these changes contributes to the variance, even when they reflect the same underlying information. The actual residual information can therefore be much lower than the bound suggests, which is consistent with the strong resistance to reconstruction we actually observe in our experiments. We believe the bound could be tightened by explicitly modeling these correlations or by accounting for exactly which attack-relevant information is retained. However, both refinements would require additional assumptions in Thm. 2 and reduce its generality. We hence leave them to future work.

(W2/Q2-2) The semantic reconstruction lower bound. The underlying reason of its nearly zero estimate is, Thm. 3 allows for the worst case for privacy, in which the attacker receives exactly the information most useful for reconstructing the target face. For example, a bit that helps the attacker determine the subject’s skin color is far more useful than a bit about the background. In this example, the bound gives the attacker the bit about skin color. In practice, some of the information the attacker obtains does little to help reconstruction. This is consistent with our findings in Sec. 4.5, where the semantic attack on InvFace is unsuccessful even though the estimated lower bound is low.

(W2/Q2-3) The pixel reconstruction lower bound. Its numerical estimate is of practical significance. The estimate was 0.23 in our original submission, which corresponds to a mean absolute error of 11.5% of the [−1,1][-1,1] pixel value range. After accounting for pixel correlations in the revision (as suggested by Reviewer eJjC), we obtain a revised estimate of 0.156. As with semantic reconstruction, the bound considers the worst case for privacy. In practice, the attack evaluated in Sec. 4.4 yields poor reconstructions.

Q3, Attack sensitivity to the identity condition: We extend the trajectory reversal experiment in original Appendix C.1 from σε=0.005\sigma_\varepsilon=0.005 to 4 different perturbation scales: 0, 0.005, 0.05, and 0.5. Notably, σε=0\sigma_\varepsilon=0 indicates the oracle setting in which the attacker is given the exact CC (the target image’s own embedding). The attack results are supplemented in Fig. 7(a)(i) of our revised main text. As OpenReview responses support only plain text, please consult the revision for visualizations. The recovered face differs from the reference, with distortions vary across settings, even under the oracle exact-CC setting. Thus, the trajectory reversal can fail even when there is no condition estimation error.

Q4, Additional comments: We appreciate the reviewer’s suggestions on paper presentation. We have clarified the distinction between visual obscurity, attack resistance, and theoretical bounds. We have also included the key numerical estimates in the revised main text.

We thank the authors for their detailed responses and revisions. The additional analysis and experiments have addressed some of our concerns and strengthened the paper.

However, some limitations remain, particularly regarding the practical informativeness of the theoretical privacy bounds and the extent to which the proposed framework provides meaningful privacy guarantees.

Overall, we appreciate the authors’ efforts and maintain our original rating of 3 (Weak Accept).

AuthorsOpenReview ↗

Thank you for your follow-up

Thank you for your follow-up and positive assessment. We appreciate your thoughtful feedback, which has helped us improve the paper. We will explore tighter and more informative privacy bounds in future work.

Reviewer_eJjCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes InvFace a privacy-preserving method that encodes a facial images into the latent space, computes a 512-dimensional identity embedding condition via a local MobileFaceNet, after that inverts the representation across deterministic DDIM steps using mirrored classifier-free guidance (mCFG) to intentionally remove visual appearance while preserving identity discriminative features or information.

Critical strengths
  • The paper uses diffusion latents directly as the object via identity-conditioned inversion without conditioning demonstrates a genuine identity-distilling mechanism.
  • The ablation trajectory across representation states (XTv→XT∅→XTmX_T^v \to X_T^\emptyset \to X_T^m) are independent, indicating the incremental gains in performance.
  • The paper proposes a semantic reconstruction attack and adapted mCFG.
Critical weaknesses
  • The paper does not present experiments with optimization-based white-box reconstruction attacks. However, if the access is given to the differentiable protection pipeline, the attacker could optimize the facial image to match the target representation, thus, avoiding the need to recover the identity-conditioning vector.
  • The unprotected baseline given in Figure 6 of the main paper, achieves only 0.51 cosine similarity, thus demonstrating that it is an evaluation bottleneck rather than a strong defense.
  • The semantic reconstruction lower bound ($4.3\times$10$^{-28}$) is tending towards zero, and the pixel reconstruction bound (D$_{rec}^{LB}$=0.23) assumes coordinate-independent entropy on pixels.
  • The client needs to run the face recognition model, namely, MobileFaceNet, to extract a 512-dimensional identity embedding “C”, it could instead output standard compact biometric templates like BioHashing or others, without running a computationally heavy 20-passes of diffusion U-Net pipeline.
  • The paper does not consider whether InvFace can fully restore the face image if an attacker has the subject’s identity embedding and access to the protection pipeline.
Questions for authors
  • The paper does not present experiments with optimization-based white-box reconstruction attacks. However, if the access is given to the differentiable protection pipeline, the attacker could optimize the facial image to match the target representation, thus, avoiding the need to recover the identity-conditioning vector.
  • The unprotected baseline given in Figure 6 of the main paper, achieves only 0.51 cosine similarity, thus demonstrating that it is an evaluation bottleneck rather than a strong defense.
  • The semantic reconstruction lower bound ($4.3\times$10$^{-28}$) is tending towards zero, and the pixel reconstruction bound (D$_{rec}^{LB}$=0.23) assumes coordinate-independent entropy on pixels.
  • The client needs to run the face recognition model, namely, MobileFaceNet, to extract a 512-dimensional identity embedding “C”, it could instead output standard compact biometric templates like BioHashing or others, without running a computationally heavy 20-passes of diffusion U-Net pipeline.
  • The paper does not consider whether InvFace can fully restore the face image if an attacker has the subject’s identity embedding and access to the protection pipeline.
3 comments
AuthorsOpenReview ↗

Response to Reviewer eJjC (1/2)

We appreciate the reviewer’s thoughtful comments. We are glad that the reviewer agree that our methodological purpose is worthwhile and that InvFace improves performance. Below, we provide several additional results and clarifications that could address the remaining concerns:

Q1, Optimization-based attack: We evaluate InvFace against two types of optimization-based attacks, both without the need to recover the identity condition, and find that InvFace is resilient to both. The results are shown in Fig. 7(b) of our revised paper. As OpenReview responses support only plain text, please consult the revision for attack visualizations.

(Q1-1) The GaFaR [1] reconstruction. The attacker reuses a pretrained EG3D [2] model, a GAN generating high-quality face images. It optimizes an MLP adapter that maps the protected latents to the GAN’s latent space. Fig. 7(b)(i) shows the attack baseline, starting from an unprotected latent. Without protection, it achieves a high average identity similarity of 0.43 to the target face (please refer to Q2 for clarification on the similarity baseline). Fig. 7(b)(ii) shows the same attack on InvFace, where the reconstruction is dissimilar to the target face. The average reconstruction similarity is as low as 0.08, suggesting InvFace’s strong resilience to the attack.

(Q1-2) Optimization from scratch [3]. The attack starts with an EG3D generation from a random GAN latent. With white-box protection access, iteratively, it obtains the InvFace-protected latent of generation and compare it with the latent of the target face. It updates the GAN latent accordingly to reduce the gap between them. Fig. 7(b)(iii) shows that this attack fails to recover the target face.

(Q1-3) Other attack forms. We further study attacks with access to the differentiable protection steps, in which the attacker either performs direct trajectory reversal (please refer to Q5), or trains a DM to recover the inversion trajectory step by step (please refer to W2 of Reviewer DrER). We find that these attacks are also unsuccessful.

[1] Comprehensive Vulnerability Evaluation of Face Recognition Systems to Template Inversion Attacks via 3D Face Reconstruction. TPAMI, 2023.

[2] Efficient Geometry-Aware 3D Generative Adversarial Networks. CVPR, 2022.

[3] Face Template Protection via Spherical Linear Interpolation. AAAI, 2025.

Q2, Semantic reconstruction similarity: In Fig. 6, the unprotected baseline’s similarity of 0.51 does not indicate an evaluation bottleneck. Instead, it is close to the actual similarity observed between real intra-person images. In reality, same-person image pairs span a wide range of similarities (rather than clustering near 1) due to variations in pose, expression, and age. Using the same similarity evaluator (a pretrained FR model), we measure the mean intra-class similarity on the CASIA-WebFace dataset, from which the attack targets are drawn. The mean is 0.52, nearly identical to the unprotected baseline. This shows that the attack is highly effective on unprotected images. Moreover, the same evaluator’s threshold for accepting a query as a match is ~0.27. Both the unprotected baseline and the compared SOTAs are well above this threshold.

AuthorsOpenReview ↗

Response to Reviewer eJjC (2/2)

Q3, Reconstruction lower bound: Thm. 3 aims to provide an analytical framework that holds for different attacks. How tight the bound is depends on what the protected representation actually reveals and how useful that information is for the specific attack.

(Q3-1) Semantic reconstruction. In Thm. 3, Eq. (22) uses UΔU_\Delta to constrain how much residual information the attacker can obtain. UΔU_\Delta considers the worst case for privacy, in which the attacker receives exactly the information most useful for reconstructing the target face. For example, a bit that helps the attacker determine the subject’s skin color is far more useful than a bit about the background. In this example, the bound implicitly assumes that the attacker is always given the skin color bit. In practice, much of the information the attacker obtains does little to help reconstruction. This is consistent with our findings in Sec. 4.5, where the semantic attack on InvFace is unsuccessful even though the estimated lower bound is nearly zero. To obtain a tighter reconstruction bound, we believe one could refine the residual information bound in Thm. 2 by accounting for exactly which information is retained. However, this would require additional assumptions in Thm. 2, reducing its generality. We leave this refinement to future work.

(Q3-2) Pixel reconstruction. That said, from Thm. 3, we can still obtain a practical numerical lower-bound estimate for pixel reconstruction. The estimate was 0.23 in our original submission, which corresponds to a mean absolute error of 11.5% of the [−1,1][-1,1] pixel value range. As the reviewer points out, this estimate could be overly ideal in treating pixel coordinates as independent. We appreciate this suggestion and account for pixel correlations; please refer to Appendix B.5 in the revision. Since these correlations are difficult to model analytically, we estimate them from real images in CASIA-WebFace and obtain a revised estimate of 0.156, which remains informative. As with semantic reconstruction, the bound considers the worst case for privacy, while the attack can yield much weaker reconstructions in practice, as evaluated in Sec 4.4.

Q4, Client-side computation cost: Using InvFace latents improves recognition accuracy and could better meet server-side security demands, compared to sending embeddings.

(Q4-1) Recogniton accuracy. Directly sending a MobileFaceNet embedding would indeed reduce client-side computation. However, this comes at the cost of recognition accuracy, which is central to FR. We benchmark the MobileFaceNet used in InvFace across LFW, CFP-FP, AgeDB, CPLFW, and CALFW. It achieves a mean accuracy of 94.73% (99.40%, 93.67%, 95.89%, 89.72%, and 94.95%, respectively), compared with 95.99% for InvFace. The additional computation thus buys 1.26% in mean accuracy over using the client embedding.

(Q4-2) Security demands. From a security perspective, InvFace’s latents, being a spatial representation, could also meet server-side sanity check demands, e.g., assessing whether the input was morphed or spoofed, for which identity embeddings provide nearly zero evidence.

Q5, White-box attack with known identity condition: In Appendix C.1 of original submission, we have considered a white-box attacker with access to the differentiable protection pipeline, the trained DM, and the unquantized latent. The attacker uses both sampling branches to reverse the noising trajectory step by step, aiming to recover the original image. Its difference to the reviewer’s proposed setting is whether the attack has the exact identity condition.

(Q5-1) Identity condition is approximated. By design, the exact condition CC is extracted from the original image by MobileFaceNet on the client and kept local. In original Appendix C.1, we assume the attacker can obtain an estimate C′≈CC'\approx C, for example, from another image of the same person. As discussed in Q2, images of the same person vary in pose, expression, and age, so CC,C′C' generally differs. To simulate such small condition mismatch, we perturbed CC with a small amount of Gaussian noise. Results in Fig.8 show that recovered images differ substantially from the targets.

(Q5-2) Identity condition is precisely given. Following the reviewer’s suggestion, we consider a worst-case oracle setting in which the attacker is given the exact CC (the target image’s own embedding, which the client does not release). We keep all other settings unchanged. As shown in Fig. 7(a)(i) of our revised paper, the attack still fails to faithfully reconstruct the target faces. It is because, even with the exact CC, each DDIM sampling update only approximately reverses the corresponding inversion update, so errors can propagate across steps.

AuthorsOpenReview ↗

Follow-up on Our Responses to Reviewer eJjC

Dear Reviewer eJjC,

Thank you for your time and constructive feedback. We have responded to your concerns and revised the manuscript accordingly. The updates include four additional optimization-based attack and trajectory reversal experiments, revised bound numerical estimates, and clarifications on reconstruction similarity and computational cost.

As the discussion period draws to a close, we would be grateful if you could take a look at our responses. We hope these revisions address your concerns and would appreciate your consideration of these updates when revisiting your assessment.

Sincerely,

Authors of InvFace

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers for their careful reading and constructive feedback. We were pleased that all the reviews highlighted InvFace’s experimental results and its approach to balancing recognition utility and visual privacy. Reviewer eJjC also noted our methodological contributions, and Reviewer E65n appreciated our information-theoretic formulation.

Across the reviews, we identified several recurring themes related to stepwise analysis, reconstruction attacks, and the numerical evaluation of the privacy bounds. To address these shared points, we made the following major revisions to the manuscript.

  1. We revised Thm. 2 to incorporate stepwise analysis explicitly and to improve its analytical capability. See Appendix B.3.
  2. We evaluated four additional reconstruction attacks. All four failed to faithfully recover the original faces. See Sec.4.6.
  3. We revised the numerical evaluation of the pixel reconstruction bound to account for correlations between pixels. See Appendix B.5.

We have also provided point-by-point responses to each reviewer. We hope that the revised manuscript and our responses address the concerns raised, and we would appreciate your taking these updates into account when revisiting your assessments.

Best,

Authors of InvFace

AuthorsOpenReview ↗

Follow-up on Our Responses to Reviewers

Dear Reviewers:

Thank you for your time and constructive feedback. We have posted our responses to your comments and revised the manuscript accordingly. As the discussion period draws to a close, could you please take a look at our responses? We hope the revisions address your concerns and would be happy to discuss any points that need further clarification.

Best,

Authors of InvFace

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.