acceptodds
Withdrawn from ICLR 2027

FaceDR-Guard for Robust Face-Swap Identity-Reuse Defense in Semantic Code Space

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Public face images can be exploited as reusable identity sources by face-swapping systems. This identity-reuse threat differs from conventional face-recognition evasion: the central question is whether a protected image can still provide identity information to a downstream generative model. We introduce FaceDR-Guard, a proactive defense that operates on the semantic code of a frozen pretrained diffusion autoencoder. For each source image, FaceDR-Guard optimizes a bounded semantic-code offset through an end-to-end differentiable surrogate pipeline comprising image reconstruction, face swapping, and identity evaluation. The optimization suppresses the source identity retained in the swap output, while skin-aware multi-scale, chromatic-consistency, pixel-fidelity, and semantic-parsing objectives regularize visual distortion. We further employ short-chain optimization followed by long-chain rendering to reduce optimization cost and improve final reconstruction quality without updating any pretrained model weights. Experiments reveal a consistent advantage under deployment shifts: FaceDR-Guard maintains strong identity-reuse suppression as conventional pixel-space attacks deteriorate, while achieving robustness competitive with feature-space cloaking and better preserving benign face-recognition utility. These findings highlight swap-output-aligned semantic reconstruction as a promising approach to robust identity-reuse defense.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
face swapping defense, proactive defense, identity reuse, diffusion autoencoder, semantic code, adversarial perturbation, facial privacy

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_aN7EOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper modifies a frozen DiffAE semantic code to produce a protected photograph and optimizes the identity retained by a downstream face-swap output. The intended use is to publish a usable photograph of the owner while reducing unauthorized identity reuse.

Critical strengths

The paper reports substantial robustness across post-processing, held-out target faces, independent recognition models, and a second face swapper. It also reports direct-recognition utility rather than only swap-output success.

Critical weaknesses

1. The central usable-photo requirement is not established.

The central use case is not convincingly satisfied: a photo owner wants to publish a usable photograph of themselves. Why would that owner publish the actual Protected x′ shown in Figure 2 (Ours row) as their own portrait? Its facial contour and eye/mouth regions change substantially relative to x; this is the published image itself, not the ×4 residual. If the owner must relinquish their original facial appearance, they could instead publish an unrelated generated face or replace their face with another identity. The paper needs to establish what useful protection it provides beyond that alternative. A low resemblance between the downstream swap and the original owner is insufficient when the identity-bearing content of the source photograph may already have been changed.

The baseline comparison must be interpreted in light of this use case. In Figure 2, Pixel-PGD and MIM appear primarily to add image-level perturbations, whereas Ours visibly changes facial geometry in this selected example. This is not proof of different human identity or population-wide failure. Table 4 actually reports higher direct ArcFace recognition failure for Pixel-PGD/MIM (100%) than Ours (61%); the corresponding failures are 81%/88% versus 15% for MobileFaceNet and 88%/92% versus 80% for MagFace. Thus, I am not claiming that the baselines have higher machine identity similarity. Their objectives explicitly attack recognition, so low x–x′ machine similarity can coexist with a photograph that remains recognizable to a human. Conversely, outperforming these baselines on that recognition-failure metric does not establish acceptable preservation of the owner's appearance. The paper should explain this distinction and assess the relevant utility directly.

Report original-to-protected identity-similarity distributions jointly with pose, expression, and human photo/identity-preservation judgments, on the same images used to measure swap defense. Use independent evaluators with clearly specified, calibrated criteria; report exact x–x′ comparisons rather than leaving cosFR's evaluation pairing/stage unclear. Table 3's PSNR 21.80, LPIPS 0.163 and other global image metrics do not resolve whether the identity-bearing face remains a usable portrait. The crucial result is defense conditioned on satisfying an original-identity and appearance-preservation requirement, together with a matched-utility comparison against the existing baselines.

2. The experiments do not isolate useful protection from source-appearance alteration.

The reported experiments do not rule out a simple alternative explanation: downstream identity suppression may be purchased by modifying the identity-bearing appearance of the source itself. Compare against the same DiffAE reconstruction without the protective modification and simple identity-altered/anonymized or face-replaced source controls at comparable photo utility. If those alternatives achieve the same effect at comparable cost, the practical benefit claimed here is not established. The present results do not prove that identity alteration explains all of the effect, but they fail to separate it from useful protection of the owner's photograph.

The different objectives used for Pixel-PGD/MIM and the proposed swap-output method further confound objective alignment, carrier choice and identity/appearance cost. Strong robustness under JPEG, resizing, blur or another swapper does not compensate for this unresolved central utility requirement.

Overall assessment. Reject. The paper does not convincingly establish its basic usable-photo protection objective. It must demonstrate a benefit over simply replacing or anonymizing the source face, while preserving the owner’s identity-bearing appearance. The reported robustness and relative machine-recognition utility do not resolve that central deficiency.

Questions for authors
  1. Who is the intended user of the published x′ in Figure 2, and why would they choose it over publishing a newly generated or face-replaced image if their original facial appearance need not be preserved? What concrete advantage is established under the same usable-photo requirement?
  2. On the same source images, can you report x–x′ identity-similarity distributions, head-pose and expression changes, and human identity/photo-utility assessments alongside swap defense? Please distinguish adversarial machine-recognition failure from visible identity change.
  3. Does the defense remain effective after restricting evaluation to x′ that satisfy a specified original-identity/appearance preservation requirement? Compare with the relevant baselines, reconstruction-only and simple identity-altered controls at matched utility.
  4. Please define Table 10's cosFR comparison pair, processing stage and evaluator precisely, and disentangle the downstream objective from the carrier under comparable preservation costs.
Reviewer_SPyYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FaceDR-Guard, a proactive defense that aims to prevent publicly shared face images from being reused as identity sources for face swapping. It perturbs the semantic code of a frozen autoencoder, optimizing it through a surrogate swapper to suppress the source identity retained in the swap output, and re-renders the protected image with the unchanged decoder.

Critical strengths

The attempt to define the defensive objective on the identity retained in the generated swap output is interesting. Differentiating through the end-to-end pipeline while keeping every pretrained model frozen is an intuitive, simple way to achieve the objective.

Critical weaknesses
  • The threat model and assumptions are not well justified. The threat model does not require the published image to still depict the owner, and fidelity is constrained only at the image level. Since protected images are often no longer matched to the owner by standard face recognition models, it is unclear whether the protection comes from disrupting the swapper or from altering the depicted identity. On the attacker's perspective, the surrogate and the swappers both draw identity from the same encoder family used for optimization, so transfer to swappers with a different identity pathway is assumed rather than shown. This shared pathway may also explain why FR-targeted baselines block the swap almost equally well, which leaves the claimed distinction from face recognition evasion unsupported. Though the attacker is further assumed to be non-adaptive, the description depends on specific FR models and unexplained fixed thresholds.

  • The reliability of references is questionable. Several references contain errors that version differences cannot explain: InfoSwap is cited under a title and venue that do not correspond to the published work, the author lineup of DiffPrivate is misnamed, and CLIP2Protect and DiffAM appear in the positioning table with incorrect venue years while being absent from the reference list. Given that the authors disclose using LLMs to retrieve related work and to cross-check references and reported numbers, these errors suggest that parts of the citations were not independently verified, which also makes the values quoted from prior work difficult to trust.

  • The advantage of the proposed design over a simple FR-targeted attack is not established. MIM, which neither models the swapper nor uses a generative model, performs comparably in various settings. The gains of the proposed method are concentrated under post-processing, where they are confounded by the magnitude of the change, and on a single additional swapper, and the paper does not explain what its considerably heavier pipeline contributes beyond these cases. Moreover, the carrier comparison is largely qualitative, and since the pixel baselines differ in both carrier and objective, the contributions of the swap-output objective and the semantic code cannot be separated. DiffProtect, which optimizes the same semantic code with a recognition objective, is not compared either.

Questions for authors
  • How much of the swap-level protection remains for protected images that are still recognized as the owner, and on what reason was the success threshold chosen?

  • Please verify the cited works and the values quoted from them, and explain why the re-evaluated baselines differ from those values.

  • What does the proposed design offer advantages compared with MIM?

  • To what extent does the persistence under common image processing depend on the size of the change rather than the perspective of the semantic code?

Reviewer_KzhUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a protection method for publicly shared face images to prevent them from being used in face-swapping attacks. Based on the DiffAE architecture, the method generates protected face images by optimizing offsets within the semantic latent space. Extensive experiments demonstrate the effectiveness of this approach in defending against face-swapping attacks across various face models, face-swapping techniques, and target images.

Critical strengths

The paper presents a very thorough experimental evaluation of defense effectiveness.

Critical weaknesses

(1) As shown in Table 3 and Figure 2, the protected image x′x' generated by the proposed method differs significantly in terms of identity from the original face image. This undermines the basis for any subsequent experiments on defense effectiveness, as the premise of the proposed method relies on the generated protected image x′x' appearing to share the same identity as the original image.

(2) The method in this paper is broadly similar to DiffProtect, but it lacks an ablation study comparing against it.

(3) The ablations in Table 10 show only marginal changes in the fidelity metrics, which seem to provide insufficient evidence for the effectiveness of the 'vis loss' used in the paper.

(4) Figure 1 includes a semantic loss term, Lsem\mathcal L_{\mathrm{sem}}, but the main text does not define it or explain how it contributes to the overall objective.

(5) The caption for Figure 4 extends beyond the boundary.

Questions for authors

Please review Q1, Q2, and Q3 under "Critical Weaknesses*."

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.