acceptodds
Under review as a conference paper at ICLR 2027

Anchoring Real Faces to Pretrained Features for Multi-Dataset Deepfake Detection

Abstract

Generative models that synthesize faces indistinguishable from real ones are proliferating rapidly. Deepfake detectors have kept pace by mining spatial and frequency artifacts, seeking cues shared across generators, and adapting large pretrained models. However, even when trained on multiple datasets, existing detectors learn which source each real face comes from: they score real faces from unseen sources closer to forgeries and reject many of them, even though their AUC remains high. In this work, we present retrained-feature-nchored earning (PAL), which anchors each training real face to its frozen pretrained feature and learns the directions along which forgeries depart from that anchor. PAL requires only binary labels and attaches to a pretrained encoder without changing its architecture. On multi-dataset protocols, PAL accepts more real faces from unseen sources than existing detectors at a comparable AUC. As a by-product, a single threshold set halfway between the score of the anchor and those of the weakest validation forgeries, without any test data, yields a higher accuracy across benchmarks than existing detectors reach even with their best single threshold. We believe PAL eases the concern over rejecting unseen real faces and, by separating the real side from the forgery side, offers a first step toward detectors that generalize to unseen forgeries.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Deepfake Detection, Multi-dataset training, Feature anchoring

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pj6xOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Pretrained-feature-Anchored Learning (PAL) for multi-dataset deepfake detection. PAL constrains real training samples to remain close to their frozen pretrained representations, while learning several forgery directions to measure how fake samples depart from this pretrained feature space. The method also introduces a fixed threshold derived from the anchor score and validation forgery scores, allowing the same decision rule to be applied across unseen datasets without test-time tuning. Experimental results show improved acceptance of unseen real faces while maintaining competitive real–fake discrimination.

Critical strengths

This paper identifies a practically important but often overlooked failure mode in multi-dataset deepfake detection: real faces from unseen sources can be misclassified as fake even when overall AUC remains high. The paper further evaluates this issue explicitly through unseen-real acceptance and thresholded accuracy, which makes the analysis more relevant to real-world deployment.

Critical weaknesses
  1. I can confirm that the issue of real samples being misclassified as fake has been studied in previous works, and that models distinguishing real data from different sources has also been studied in previous works. Then, is the real novelty of this paper merely to make the representations of real data closer to the pretrained representations?

  2. Keeping the representations of real data unchanged does not necessarily mean that it is beneficial for real–fake classification.

  3. Source-domain real data still participate in training. This method mainly constrains source-domain real samples to remain close to their pretrained representations, but this cannot guarantee that the actual decision logic of the model changes. Since the fake representations and the classification boundary can still adaptively adjust, the model may simply relearn a decision rule similar to the original one in a new representation coordinate system. Therefore, the current method is more like re-constraining the reference position of the representation space rather than substantially optimizing the decision mechanism. The authors even acknowledge that “whereas training reals can only be shifted slightly against them within the anchor’s slack”.

  4. Under this training scheme, the most questionable issue is whether test real data can still remain consistent with their frozen features. If not, the method would be meaningless.

  5. Overall, this work proposes an interesting but rather flawed idea, and the method design cannot solve the problem it raises.

Questions for authors

If the authors can theoretically demonstrate that their method truly changes the detector’s decision logic, I believe this would make the work more complete and would improve my score. In its current form, the method essentially maintains eight forgery-class prototypes while constraining the real distribution to remain unchanged during training, which I find difficult to be convinced by.

2 comments

I thank the authors for their response. After considering it, I have decided to maintain my original score.

We thank the reviewer for reading our rebuttal and for the response.

In our rebuttal, we aimed to address each of the concerns raised in the initial review in detail. However, we would be grateful if the reviewer could clarify which concerns remain unresolved, or which parts of our response were not convincing. We would also greatly appreciate any specific suggestions, such as additional experiments, analyses, or clarifications, that would strengthen the paper, and we will do our best to address them.

Reviewer_tLHcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Pretrained-feature-Anchored Learning (PAL) for multi-dataset deepfake detection. The authors argue that detectors trained on multiple datasets can learn source-specific characteristics of real faces and consequently reject genuine faces from unseen sources, even when their overall AUC remains high. PAL addresses this issue by anchoring the features of training real faces to those produced by a frozen pretrained encoder, while learning forgery evidence from the feature departures induced by fine-tuning. A scoring head projects these departures onto a small set of learned forgery directions. The authors also introduce a thresholding rule based on the score of zero feature departure and the lower tail of validation-forgery scores. Experiments across several multi-dataset training settings and unseen-source evaluation benchmarks show that PAL accepts more genuine faces from unseen sources than competing detectors at selected operating points, while maintaining competitive real–fake discrimination. The paper also evaluates PAL with different pretrained encoders and as an addition to existing detection methods.

Critical strengths
  1. The paper identifies a relevant problem in deepfake detection. Overall AUC does not reveal whether a detector rejects a substantial proportion of genuine faces from unfamiliar sources. Evaluating genuine-face acceptance separately from forgery detection is useful, particularly for deployment settings where false accusations against authentic media are costly.

  2. The proposed design is reasonably simple and motivated by the problem. Anchoring training real features, while allowing fake features to move during fine-tuning, provides an intuitive way to reduce reliance on the limited real sources seen during training. The experiments with different encoders also suggest that the approach is not restricted to a single backbone.

Critical weaknesses
  1. The central explanation for the observed failure is plausible, but not directly established. The paper argues that detectors reject unseen real faces because they exploit source-specific fingerprints. Results in A.3 demonstrates that source-specific information is recoverable from face images, but it does not establish that the evaluated binary detectors actually rely on these signals when making real–fake decisions. The observed rejection of unseen real faces could also arise from other aspects of distribution shift, including image quality, acquisition pipelines, or differences in facial content. A direct analysis of the baseline representations or controlled interventions on source-specific cues would make the proposed diagnosis much more convincing. At present, the motivation for PAL rests partly on an inference that is not adequately tested.

  2. Anchoring training real features does not guarantee that unseen real faces will remain on the real side of the decision boundary. The anchor loss penalizes the feature displacement of individual training real samples, while the scoring head classifies inputs according to their displacement along learned forgery directions. Neither component directly constrains the displacement distribution of genuine faces from unseen sources. Such faces may respond differently to fine-tuning because their acquisition characteristics, identities, or image quality differ from those of the training reals. The paper's own degradation experiment illustrates this limitation: under stronger combinations of blur, noise, and compression, PAL loses its advantage in real acceptance, with FSFM and Effort accepting more genuine faces at the more severe levels. This is an important limitation because preserving acceptance of unfamiliar real faces is the main motivation for the method, rather than a secondary consideration.

  3. The data sampling procedure does not consistently balance training exposure across sources. According to Appendix A.1.4 and Table 8, PAL uses class-balanced batches but samples frames uniformly within each class; it does not use the source-balanced sampling employed by several compared methods. The fake sources are capped individually, whereas real frames are not capped in the same way. Consequently, large real datasets can contribute disproportionately to the anchor objective. This matters directly for a method whose central motivation is to reduce dependence on particular real sources. If the anchor is dominated by a subset of the available real data, it is unclear how much the method is learning a broadly transferable notion of real facial content rather than preserving the features of the most frequently sampled sources. A source-balanced version of PAL would be an important control, both for evaluating the method itself and for ensuring fair comparisons with baselines that use source-balanced sampling.

Questions for authors

See weakness

Reviewer_pg7QOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a deepfake detection method that aligns real samples with pre-trained features, and uses departures from pre-trained features as a criterion for identifying fake samples. For real samples, the model is trained to minimize its departure from original features. Further, the method selects K prior directions of departure and uses alignment to the directions as scores for classification. An automatic threshold is selected according to the proposed geometry of the feature space. Three new experiment settings are proposed where the models are trained on multiple datasets, and real sample accuracy are shown to greatly improve. Overall improvements over real and fake samples are moderate but consistent across backbones.

Critical strengths
  • The paper makes a reasonable call to move away from single-source training and evaluate models trained on multiple datasets.
  • The automatic threshold also distinguishes the paper from existing work, which primarily emphasize AUC and could hinder adaptations to unseen distributions.
  • PAL achieves AUC comparable to existing work while improving on accuracy with automatically selected threshold.
  • Comparison with WiSE and Effort highlights the advantage of the asymmetrical processing of real and fake samples.
Critical weaknesses
  • Selecting K prior directions for fake samples seems to be inconsistent with the overall goal to generalize to unseen samples. Considering that real samples are constrained by its distance to original features, could it better to set s(x) also according to distance, instead of direction alignment with priors?
  • PE-Core appears to be a strong model even as a bare backbone. Improvement over the bare backbone is shown consistent across different backbones, though relatively limited.
  • Some cases have seen improved real sample accuracy at the expense of very low fake sample accuracy, especially with out-of-domain data. This is expected with a comparable AUC and is acknowledged in discussion, but it suggests that a more balanced design for the automatic threshold is needed.
  • Thresholding strategy could be discussed more in depth. For example, could Q_0.05 allow outliers in the validation set to disproportionately affect the threshold?
Questions for authors
  • Have other threshold selection rules been explored, and what is the selection criterion?
  • What is a typical size of the validation set, and what does the distribution of scores look like? Does Q_0.05 account for potential outliers, which could affect the threshold?
  • Does Table 6 account for differences in backbones?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.