acceptodds
Under review as a conference paper at ICLR 2027

Swapped Face, Same Person: Measuring the Identity That Face Swaps Leave Behind

Abstract

Face-swap methods report identity transfer as ArcFace similarity on a tight face- oval crop, and a low crop-similarity to the target is read as evidence that the target was de-identified. This practice is blind by construction: the region it discards—hair, ears, skull contour—is exactly the region a face-oval swapper never modifies. The general failure is that the metric is evaluated only on the support of the transformation, so any residual identity channel outside that support goes unmeasured; face swapping is the case where the gap is widest, but the construction is not specific to it. We localize that residual identity with segmentation-defined probes, replacing the arbitrary widened bounding box used in prior work with CelebAMask-HQ head regions, and in particular with a pericranial ring probe that sees only the pixels a face-oval swapper copies verbatim from the target. Because such regions have a much higher impostor-similarity floor than aligned face crops (mean 0.447 vs. 0.005 on our gallery), raw cosines are not comparable across probes, so we report calibrated rates instead: thresholds are set per probe at a stated false-accept rate on identity-disjoint impostor pairs, following the attacker- aware protocol of recent encoder-privacy work. At , the target is accepted from hair and ears alone for 57–97% of the outputs of the face-oval swappers shipped in a widely deployed open-source toolkit, against 0.3–61% under the standard crop; equal-area, equal-strength occlusion attribution confirms this is a property of which region is read, not of how many pixels it contains. The measurement is also discriminative: a full-head regeneration teacher that rewrites the pericranial region leaks 2.3% under the same probe, a 13–42× separation from every face-oval method that no face-crop metric reproduces. We release PHANTOM-150K’s generation pipeline, per-sample metadata, and all evaluation code—but not generated images, since every one depicts real donor and target identities.

Primary area:
datasets and benchmarks
Keywords:
Face swapping, face manipulation, face synthesis, deepfakes, generative AI, diffusion models, synthetic data, face datasets, facial identity, facial expression, identity preservation, expression preservation, privacy-preserving face manipulation, face forgery detection, deepfake detection, face forensics, multimedia forensics, computer vision, image generation, image-to-image translation, generative models, foundation models, dataset curation, benchmark dataset, FLUX, full-head face swapping, facial recognition, identity leakage, privacy-aware evaluation, synthetic face images, deepfake datasets

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fF8TOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that standard ArcFace face-crop evaluation overlooks identity cues in unchanged regions such as hair and ears. It introduces segmentation-based face, head, and ring probes with separately calibrated thresholds, using PHANTOM-150K full-head swaps as a control. At a 0.1% false-accept rate, the ring probe matches the original target in 29–97% of face-oval swaps, compared with 2.3% for the full-head teacher. The authors propose evaluating privacy-preserving swaps through both successful donor-identity transfer and the absence of remaining target identity.

Critical strengths

• The paper identifies an important evaluation gap. Passing a test on the central face does not tell us whether the surrounding hair and ears still match the original image.

• The measurements are carefully checked. The authors use separate thresholds for each region, statistical tests, identity-separated comparisons, and checks for changes in region boundaries.

• The reporting is open and detailed. The authors report rejected samples, provide information needed to reproduce results, explain baseline limitations, and correct their earlier bounding-box test.

Critical weaknesses

• The ring test may confuse copied pixels with identity, because Figure 4 shows that the surrounding region remains almost unchanged, so high similarity does not establish whether the probe recognizes the person or simply matches the original image.

• The paper never tests whether the ring can match separate images of the same person, leaving its usefulness under changes in lighting or hairstyle unproven.

• The gallery may contain the exact target photographs used to generate the swaps, allowing high accuracy through matching unchanged regions rather than identifying someone from another photograph. So, the retrieval results are difficult to interpret.

• The paper does not document who claims successful identity removal from face-crop similarity or discuss relevant anonymization research, making the scope of the problem it criticizes unclear.

• The contribution beyond existing methods needs clearer justification, because calibration, retrieval, region definitions, and the generation adapter come from prior work, leaving it unclear what new capability or insight their combination provides.

• The full-head comparison does not clearly establish an independent advantage, because BFS-Qwen shares the teacher’s adapter family and uses different evaluation pairs, galleries, and thresholds, so its lower leakage rate cannot be directly attributed to a better method.

• The proposed privacy rule does not assess all measured identity leakage, because Equation 17 considers only the ring region and ignores leakage within the central face, potentially accepting a result that still reveals the target’s identity.

• The evaluation is too limited to support broad benchmark claims, because the reported experiments use only 300 pairs, include several limited baselines, and omit FaceFusion/inswapper from most supporting analyses.

Questions for authors

• Does the gallery contain the exact target photographs used to generate the swaps, and what happens when those photographs are replaced with different images of the same people?

• How often does the ring test correctly match different photographs of the same person at a false-match rate of 0.1%, and how does a hairstyle change affect this?

• How do simple image-comparison methods, such as SSIM or perceptual hashing, perform, and what happens when the ring’s shape is preserved but its hair content is replaced?

• Can you cite a paper or system that claims identity removal based only on face-crop similarity, and explain how your work differs from DeepPrivacy, CIAGAN, FALCO, RiDDLE, and person re-identification research?

• Why is the independent full-head method by Jeong et al. (2025) not evaluated, even though it helps motivate your approach?

• Is BFS-Qwen-V4 evaluated using the same 300 pairs and thresholds as the other methods, and, if not, how should its 1.7% result be compared with theirs?

• How do the face-oval methods perform on image pairs rejected by the quality filter that was tuned using the teacher model?

• Why does Equation 17 consider only the ring region, and how would the conclusions change if identity leakage across all tested regions were considered?

• Can you explain how the Spearman p-value was calculated and include uncertainty in the ring threshold when reporting leakage rates?

• Why is FaceFusion/inswapper missing from Tables 3, 4, 7, 8, and 9, and can its results be added?

• Why could 994,293 image-pair records not be loaded, and could the missing examples differ in ways that affect the evaluation?

Reviewer_LX2pOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that conventional face-swap evaluation can underestimate residual target information because identity similarity is typically measured on an aligned face crop, whereas common face-oval swappers preserve the target's hair, ears, headwear, and surrounding head structure. The authors propose segmentation-defined face, head, and pericranial-ring probes, with probe-specific thresholds calibrated at a specified FAR, and evaluate both donor retention and target leakage. They further introduce PHANTOM-150K, use full-head regeneration as a negative control, perform equal-area occlusion analysis, test multiple representations, and evaluate an independently generated full-head system. The main empirical finding is that the pericranial probe matches face-oval outputs to their paired target image substantially more often than the conventional face crop. The paper therefore exposes a meaningful evaluation blind spot, although the current experiments establish exact-target-image linkage more convincingly than person-level biometric identity leakage.

Critical strengths
  1. The conventional aligned face crop evaluates almost exclusively the region modified by face-oval swappers, while ignoring unchanged regions that may retain target-related information. The segmentation-defined probes and full-head controls make this failure mode considerably more explicit than a simple enlarged bounding-box comparison.

  2. The authors correctly show that raw cosine similarities from aligned face crops and pericranial regions are not directly comparable because their impostor distributions differ substantially. Calibrating each probe at the same nominal FAR, together with the equal-area analysis, cross-representation experiment, and independent full-head control, addresses several important confounders.

Critical weaknesses
  1. For face-oval methods, the pericranial region largely consists of pixels copied from the original target image. In the verification experiments the output is compared with that same target image, and the gallery-retrieval experiment also appears to include the exact generating target photograph. Consequently, the very high ring retrieval rates may primarily measure near-duplicate image matching rather than recognition of the target person from an independent observation. This distinction directly affects the paper's central claims about "identity," "de-identification," and finding the "protected person." A same-person, different-image experiment with the generating photograph excluded is needed to support the stronger biometric interpretation.

  2. Section 4.3 states that true identities are not directly observable and therefore removes pairs with standard-crop similarity above 0.50 as presumed same-identity pairs. This is a heuristic impostor-cleaning procedure, not identity-disjoint calibration. In addition, thresholds are estimated from many dependent pairwise comparisons of a relatively small gallery, while uncertainty in the estimated threshold is not propagated into the leakage confidence intervals. These issues weaken the interpretation of the reported 10−310^{-3} operating point.

Questions for authors
  1. Can the authors evaluate target leakage using a different image of the same target person, with the original target photograph used for generation completely excluded from both verification templates and retrieval galleries? This experiment would determine whether the reported pericranial effect represents biometric person-level identity leakage rather than exact-image or copied-pixel linkage. The answer would materially affect my assessment.

  2. Please clarify the construction of the "600-identity gallery" used in Table 9. Does the gallery contain the exact target image used to generate each swap? If so, I believe the experiment should be described as target-image retrieval rather than person re-identification. If independent images of the same identities are used, please specify the identity labels and gallery construction.

  3. Please report results with genuinely identity-labelled, identity-disjoint calibration subjects, or at least quantify sensitivity to the sstd>0.50s_{\mathrm{std}}>0.50 presumed-same-identity exclusion rule? The current description appears inconsistent with the repeated claim of identity-disjoint FAR calibration. Flag for Ethics Review

Reviewer_QWmBOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that the standard way of evaluating face swap privacy is blind to an obvious leak and proposes a better measuring technique. The real problem with existing measurements is that they only replace the inner face oval while hair, ears, and skull outline remain original. The ArcFace evaluation pipeline throws away everything outside the tight face crop and verifies that the swap is "de-identified," while the target's whole head and ears are still sitting there to be identified. The paper takes inspiration from this and develops a new metric that includes this leak in the new measure. The paper introduces segmentation defined probes and calibrates separate thresholds for each imposter pair. The paper also claims a phantom 150k dataset which is triplet full head swap corpus.

Critical strengths
  • The paper showcases a new measure that includes leakage which the conventional measure was not able to measure. This is much needed in the domain of privacy and de-identification for its advancement.
  • The paper provides artifact hashes, seeds as pure functions of pair keys, a full funnel accounting including rejections, compute and energy reporting.
  • The paper recognizes that a probe which reads high everywhere proves nothing. It constructs a negative case that the probe should read as low, and it does. This is better experimental logic than most benchmark papers show.
Critical weaknesses
  • Paper feels a bit unclear at multiple points such as
    • Paragraph 3 (“Our earlier attempt... used a widened bounding box... both attacks are correct”) assumes the reader knows the earlier submission and its reviews.
    • The admission that ϕring\phi_{ring} is “closer to copy detection” appears in the intro’s “What is and is not claimed” paragraph, again in Limitations, and again in Appendix D and each time it’s phrased slightly differently.
    • The title, abstract and contribution list present PHANTOM-150K as a major contribution, but Section 3 opens by calling its role “instrumental rather than headline.” which is confusing.
    • Section 3 already cites the key result before the probes or calibration have been defined in Section 4.
  • No trained diffusion swapper is tested, for example Jeong et al.’s head-swapping method is cited as motivation but never evaluated, even though it would make a natural independent full-head comparison.
  • “Identity survives outside the face”is already established in [1] which paper does not cite or talks about it.

[1] Hukkelås, H., & Lindseth, F. (2023, January). Deepprivacy2: Towards realistic full-body anonymization. In 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (pp. 1329-1338). IEEE.

Questions for authors
  1. Is the gallery 600 identities or 600 images? Section 5.3 says “600-identity gallery”; Appendix B says “600-image gallery.” Those mean different things for both calibration and retrieval.
  2. Why is FaceFusion missing from most supporting tables? It’s the paper’s headline example, but it appears only in Table 1. Tables 3, 4, 7, 8 and 9 cover the other five methods only.
  3. Which machine is “here” and which is “the second machine”? The Table 1 footnote says FaceFusion was “run here” while the others came from the second machine. Does that affect comparability?
  4. Why do Table 2’s radius-0 values differ from Table 1? For example, SimSwap is 72.7% vs. 69.3% and BFS-FLUX 3.3% vs. 2.3% for what should be the unperturbed ring.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.