Anchoring Real Faces to Pretrained Features for Multi-Dataset Deepfake Detection

Generative models that synthesize faces indistinguishable from real ones are proliferating rapidly. Deepfake detectors have kept pace by mining spatial and frequency artifacts, seeking cues shared across generators, and adapting large pretrained models. However, even when trained on multiple datasets, existing detectors learn which source each real face comes from: they score real faces from unseen sources closer to forgeries and reject many of them, even though their AUC remains high. In this work, we present $\text{P}$retrained-feature-$\text{A}$nchored $\text{L}$earning (PAL), which anchors each training real face to its frozen pretrained feature and learns the directions along which forgeries depart from that anchor. PAL requires only binary labels and attaches to a pretrained encoder without changing its architecture. On multi-dataset protocols, PAL accepts more real faces from unseen sources than existing detectors at a comparable AUC. As a by-product, a single threshold set halfway between the score of the anchor and those of the weakest validation forgeries, without any test data, yields a higher accuracy across benchmarks than existing detectors reach even with their best single threshold. We believe PAL eases the concern over rejecting unseen real faces and, by separating the real side from the forgery side, offers a first step toward detectors that generalize to unseen forgeries.