Swapped Face, Same Person: Measuring the Identity That Face Swaps Leave Behind

Face-swap methods report identity transfer as ArcFace similarity on a tight face- oval crop, and a low crop-similarity to the target is read as evidence that the target was de-identified. This practice is blind by construction: the region it discards—hair, ears, skull contour—is exactly the region a face-oval swapper never modifies. The general failure is that the metric is evaluated only on the support of the transformation, so any residual identity channel outside that support goes unmeasured; face swapping is the case where the gap is widest, but the construction is not specific to it. We localize that residual identity with segmentation-defined probes, replacing the arbitrary widened bounding box used in prior work with CelebAMask-HQ head regions, and in particular with a pericranial ring probe that sees only the pixels a face-oval swapper copies verbatim from the target. Because such regions have a much higher impostor-similarity floor than aligned face crops (mean 0.447 vs. 0.005 on our gallery), raw cosines are not comparable across probes, so we report calibrated rates instead: thresholds are set per probe at a stated false-accept rate on identity-disjoint impostor pairs, following the attacker- aware protocol of recent encoder-privacy work. At $FAR=10^{-3}$, the target is accepted from hair and ears alone for 57–97% of the outputs of the face-oval swappers shipped in a widely deployed open-source toolkit, against 0.3–61% under the standard crop; equal-area, equal-strength occlusion attribution confirms this is a property of which region is read, not of how many pixels it contains. The measurement is also discriminative: a full-head regeneration teacher that rewrites the pericranial region leaks 2.3% under the same probe, a 13–42× separation from every face-oval method that no face-crop metric reproduces. We release PHANTOM-150K’s generation pipeline, per-sample metadata, and all evaluation code—but not generated images, since every one depicts real donor and target identities.