- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes InvFace, a privacy-preserving face recognition method based on identity-distilling diffusion inversion. Unlike conventional single-step representation transformations, the authors introduce a stepwise privacy-preserving paradigm consisting of multiple explicit transformation steps. By leveraging the inversion process of an identity-conditioned diffusion model, the proposed method generates protected representations that preserve identity recognition capabilities while concealing visual information.
Critical strengths
The paper explores the use of identity-conditioned diffusion inversion for privacy-preserving face recognition. By combining identity-distilling inversion, mirrored classifier-free guidance (mCFG), and quantization, the authors aim to achieve a balance between identity recognition performance and visual information concealment.
Critical weaknesses
-
The paper attributes its main contribution to privacy quantification to sequential small-step transitions, arguing that explicit stepwise transformations enable the analysis and accumulation of information changes across individual stages, thereby providing a computable theoretical upper bound on residual privacy leakage. However, if the same form of information upper bound can be obtained for an arbitrary monolithic protected representation under the same quantization setting by directly estimating its terminal within-identity variance, the authors need to further clarify what additional theoretical properties, analytical capabilities, or tighter privacy bounds the proposed stepwise paradigm offers for privacy quantification.
-
The paper employs mCFG to alter the diffusion inversion trajectory, preventing the protected representation from being directly recovered through the original unconditional or conditional sampling branches. However, the inability of the original sampling branches to directly reverse the mCFG trajectory does not necessarily imply that a recovery model specifically trained or optimized to reverse mCFG would also fail.
Questions for authors
-
The within-identity variance statistic used in Theorem 2 ultimately equals the terminal within-identity variance (Vr), since the variance and covariance recursions across the transformation steps telescope exactly to this terminal statistic. Could the same form of information-theoretic upper bound therefore be obtained for an arbitrary monolithic protected representation under the same quantization setting simply by directly estimating its terminal within-identity variance? If so, could the authors further clarify what essential theoretical advantages the proposed stepwise paradigm provides for privacy quantification over directly analyzing the terminal representation, beyond merely decomposing the same terminal statistic into multiple steps?
-
Table 4 reports an information leakage upper bound that is close to the maximum information capacity of the quantized representation, while Table 5 reports a semantic reconstruction error lower bound that is nearly zero. Could the authors clarify how these results can be interpreted as meaningful privacy guarantees?
-
Theorems 2 and 3 provide an upper bound on the conditional mutual information of the protected representation and a corresponding lower bound on reconstruction error, respectively. However, whether these theoretical results provide meaningful quantitative guarantees regarding practical privacy risks remains to be further validated. Specifically, Table 4 reports an information upper bound of approximately 7.37 bits per representation coordinate, whereas the protected representation uses 8-bit quantization. Since this upper bound is relatively close to the maximum information capacity of the quantized representation, it places only a relatively loose constraint on residual information leakage. Therefore, although the proposed theoretical framework provides computable information-theoretic expressions, the paper has not sufficiently demonstrated that these quantitative results can effectively characterize differences in practical privacy risks across different protection configurations and PPFR methods.
4 comments
Response to Reviewer DrER (2/2)
W2, trained model for trajectory reversal: We study a recovery model specifically trained to reverse mCFG.
(W2-1) Trajectory reversal with DM. We consider a white-box attacker with full knowledge of the step-wise protection pipeline and access to both the protected latent and the exact identity embedding . The attacker trains a diffusion model on pairs of protected latents and their corresponding original images, using the protected latent as the initial noisy state and the original image as the generation target. At each recovery step, the model takes the same current latent and computes two updates, one without and one conditioned on , and combines the two updated latents using mCFG. The attack result is shown in Fig. 7(a)(ii) of our revised main text. As OpenReview responses support only plain text, please refer to the revision for visualizations. Even with targeted training and access to the exact identity embedding, the attacker DM failed to faithfully reconstruct the original faces.
(W2-2) Other types of attacks. We have also supplemented our evaluation with other types of attacks to more comprehensively assess the protection offered by InvFace, including two optimization-based attacks and direct trajectory reversal given the exact . InvFace is resilient to all the attacks studied. If the reviewer is interested in these analyses, please refer to our responses to Reviewer eJjC’s (Q1-1), (Q1-2), and (Q5-2).
Q2/Q3, Informativeness of numerical estimates: We provide a clearer interpretation of the numerical estimates of bounds below.
(Q2/Q3-1) The information upper bound. We acknowledge that the estimate of 7.23 bits/coordinate is close to the 8-bit capacity ceiling. That said, we note that this does not imply a privacy deficiency and that the bound can be further tightened. From a privacy perspective, since quantization itself is a protection step, the estimate should be read as showing how much inversion further tightens the bound, with the quantized latent serving as the reference point instead of an unprotected baseline.
The bound is also conservative in its current form. One source of looseness is that it uses pixel-wise variance without accounting for correlations between pixels. For example, a single change in facial pose can affect many pixels at once. Each of these changes contributes to the variance, even when they reflect the same underlying information. The actual residual information can therefore be much lower than the bound suggests, which is consistent with the strong resistance to reconstruction we actually observe in our experiments. We believe the bound could be tightened by explicitly modeling these correlations or by accounting for exactly which attack-relevant information is retained. However, both refinements would require additional assumptions in Thm. 2 and reduce its generality. We hence leave them to future work.
(Q2/Q3-2) The semantic reconstruction lower bound. The underlying reason of its nearly zero estimate is, Thm. 3 allows for the worst case for privacy, in which the attacker receives exactly the information most useful for reconstructing the target face. For example, a bit that helps the attacker determine the subject’s skin color is far more useful than a bit about the background. In this example, the bound gives the attacker the bit about skin color. In practice, some of the information the attacker obtains does little to help reconstruction. This is consistent with our findings in Sec. 4.5, where the semantic attack on InvFace is unsuccessful even though the estimated lower bound is low.
(Q2/Q3-3) The pixel reconstruction lower bound. Its numerical estimate is of practical significance. The estimate was 0.23 in our original submission, which corresponds to a mean absolute error of 11.5% of the pixel value range. After accounting for pixel correlations in the revision (as suggested by Reviewer eJjC), we obtain a revised estimate of 0.156. As with semantic reconstruction, the bound considers the worst case for privacy. In practice, the attack evaluated in Sec. 4.4 yields poor reconstructions.
(Q2/Q3-4) Numerical estimates for configuration choice and SOTA comparison. The estimated information upper bound provides an analytical basis for choosing protection configurations. As shown in W1/Q1, the stepwise analysis explains how the bound varies across inversion endpoints and guides the choice of . Since the evaluated PPFR SOTAs do not report comparable numerical privacy bounds, we compare their protection experimentally through the pixel and semantic reconstruction experiments in Secs. 4.4 and 4.5.
Thank you for the detailed rebuttal and the additional experiments targeting mCFG. I appreciate these efforts, which further strengthen the paper's experimental evaluation. In addition, the authors have provided further theoretical derivations in the revised manuscript, improving the interpretability of the stepwise analysis. Overall, the rebuttal has addressed most of my previous concerns.
Thank You for Your Follow-Up
Thank you for your thoughtful and positive follow-up. Your acknowledgment of our efforts means a lot to us. We are glad that our rebuttal has addressed most of your concerns. In light of these improvements, would you be willing to consider raising your rating?
Many thanks!
Response to Reviewer DrER (1/2)
We appreciate the reviewer’s helpful comments and thank them for highlighting InvFace’s approach to balancing recognition utility and visual privacy. Below, we provide additional results and clarifications to address the reviewer’s concerns.
W1/Q1, Stepwise analysis of information upper bound: We agree that the information upper bound in the original submission can be obtained monolithically and that the additional analytical capabilities need further justification. We have revised Thm. 2 to explicitly incorporate stepwise analysis, which also gives us an analytical basis for choosing the endpoint T.
(W1/Q1-1) Stepwise derivation of Thm. 2. We here provide a proof sketch; please refer to Appendix B.3 of our revised paper for the full derivation. In the original derivation, summing the stepwise variance changes recovers the same variance that can be measured directly at the endpoint. In the revision, we consider two images of the same person. Their latent states and identity conditions can both differ. For the t-th step Ft, we first change the state while holding the condition fixed, then change the condition while holding the new state fixed:
δtst=Ft(z,C)−Ft(z′,C),δtcond=Ft(z′,C)−Ft(z′,C′).
Their sum is the actual difference between the two outputs. Squaring and averaging this difference gives a state term, a condition term, and a signed cross term. Using the RMS state sensitivity ρtst, the condition contribution χt, and the positive part of the normalized cross term ηtsc, we propagate the variance budget as
Bt=(ρtst)2Bt−1+χt2+2[ηtsc]+ρtstBt−1,χt,B0=V0.
Taking the positive part makes the update nondecreasing in the preceding variance, so the recursion preserves the upper bound at every step. We then use the same quantization and entropy argument to obtain UΔ from BT. The stepwise analysis adds a diagnostic capability: it shows where changes in within-identity variance occur and how state propagation, condition variation, and their interaction contribute to those changes.
Based on the new derivation, using real samples from CASIA-WebFace, we estimate the revised upper bound at 7.23 bits/coordinate, which is comparable to the original estimate.
(W1/Q1-2) Choice of the inversion endpoint. The revised derivation of Thm. 2 also helps us choose the endpoint T. As inversion progresses, the latents become more noise-like and image-specific facial details become harder to discern. Intuitively, one might expect further inversion steps (increasing T) to always tighten the information upper bound. However, both original and revised numerical estimates reveal a sweet point, where the bound reaches its minimum at around T=500 and rises at T=550.
The stepwise recursion helps explain this reversal. With the mCFG weight fixed, each update propagates the differences already present in the latent state, introduces differences due to the identity condition, and allows the two to interact. Subtracting the preceding variance budget from both sides of the recursion gives
\underbrace{\big[(\rho_t^{\mathrm{st}})^2 - 1\big]\mathcal{B}_{t-1}}_{\text{state contraction term}} + \underbrace{\chi_t^2 + 2[\eta_t^{\mathrm{sc}}]_+ \rho_t^{\mathrm{st}} \sqrt{\mathcal{B}_{t-1}}\\,\chi_t}_{\text{condition and cross terms}}.$$ Even if an update (increase of $T$) continues to contract the state, a similar contraction coefficient removes less variance once the preceding variance budget is smaller. The condition and cross terms can then outweigh that reduction. This is what we observe around the turning point. At $T=500$, state contraction outweighs the condition and cross terms (0.0137 vs. 0.0151 by magnitude). At $T=550$, the condition and cross terms are larger and outweigh state contraction (0.0284 vs. 0.0133). This explains the increase in the propagated variance budget at that update and supports choosing $T=500$, where the estimated bound is lowest.