acceptodds
Under review as a conference paper at ICLR 2027

What a Deepfake Score Does Not Tell You: Same-Source Controls for Face-Forgery and Generated-Video Detectors

Abstract

Real/fake performance alone does not determine how a detector responds to non-generative processing, neural re-rendering, or cross-identity conditioning. We introduce source-matched controls for face-forgery and generated-video evaluation. For 136 FaceForensics++ test excerpts and two face-swap generators, we retain each original (R) and construct a pipeline-only arm (N), a self-swap (S), and a cross-identity swap (F) with the same implemented downstream pipeline. Across 29 released checkpoints from five families, several assign high forgery scores to pipeline-only videos (N/R AUC up to 0.97; at a common 0.5 threshold two video pipelines judge 63–65% of them fake against 1% of originals). The conditional discrimination between cross-identity and self-swap outputs (F/S) varies across the 26 primary checkpoints (0.47–0.79) and is weaker than the conventional forgery-versus-original discrimination for every one of them. On a third generator, GHOST, absent from the documented detector-training methods, N/R exceeds 0.5 for all 26 primary checkpoints and F/S is below F/R for 25. On FaceForensics++ test crops at threshold 0.5, Gaussian blur (σ=4) turns 99–100% of initially correctly classified real frames into false alarms for 16 of the 19 frame-level checkpoints, and JPEG quality 30 turns 15–96% of correctly detected NeuralTextures frames into misses for the 12 FaceForensics++-trained ones. In a controlled Xception system, pipeline-matched negatives suppress the positive pipeline response. With fixed SimSwap four-arm inputs and five epochs, changing only the self-swap target raises F/S from 0.55 to 0.97 on the training generator and from 0.56 to 0.69 on GHOST (not confirmed on InSwapper), and a shared-backbone two-head model supports both targets. A source-matched generated-video case study shows detector- and implementation-dependent responses to format conversion and same-family VAE reconstruction. Standard AUC does not resolve these operational sensitivities; we recommend reporting matched non-generative controls, conditional contrasts, and the inference and control-construction settings alongside detection results.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
deepfake detection, AI-generated video detection, evaluation methodology, same-source controls, face forgery, video forensics, video compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xm1xOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces source-matched original, pipeline-only, self-swap, and cross-identity-swap controls to characterize deepfake detector responses. Evaluation of 29 checkpoints shows that strong conventional detection AUC can coexist with sensitivity to non-generative processing and weak cross-versus-self-swap discrimination. Controlled Xception training interventions and a generated-video case study demonstrate how supervision and processing settings affect these responses.

Critical strengths

1.The matched controls expose behavior that conventional real/fake AUC leaves unresolved. Explicitly distinguishing synthesis detection from identity discrimination makes the protocol useful and its interpretation careful.

2.Broad checkpoint coverage, native inference pipelines, compression checks, and tightly matched training interventions provide substantial evidence for the central diagnostic claims.

Critical weaknesses

1.Evidence for practical training improvements is limited to one Xception system. Gains transfer unevenly across generators, and the intervention models perform near chance on standard FaceForensics++ manipulations. This limits conclusions about improving broadly useful detectors.

2.Construction code, inference harnesses, and predictions are unavailable during review. Because the results depend on precise processing choices, this limits independent verification of the central empirical contribution.

Questions for authors

1.Does the self-swap relabeling gain persist when effective class balance and loss contributions are controlled? Relabeling also halves the positive fraction, leaving its contribution unresolved.

Reviewer_QwknOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper tackles a really important blind spot in deepfake detection: standard evaluation metrics (like real-vs-fake AUC) just lump processing, neural rendering, and identity changes all together. To fix this, the authors propose a clever same-source four-arm protocol (R,N,S,FR, N, S, F) that properly isolates these factors. After testing a massive lineup of 29 checkpoints and 3 generators (plus a held-out one, GHOST), they show that many detectors actually flag normal processing pipelines as fake, and most struggle to reliably tell a self-swap apart from a cross-identity swap. Overall, it's a super thorough, eye-opening study that challenges how we benchmark these models.

Critical strengths

Breaking down the standard comparison into pipeline-only (NN), self-swap (SS), and cross-identity (FF) arms is a brilliant way to unmask what detectors are actually picking up on.

The authors didn't just test a couple of models; they evaluated 29 checkpoints, multiple post-processing degradations (blur, JPEG compression), and even extended the framework to modern generative video models (like CogVideoX1.5 and LTX-Video).

Critical weaknesses

1.The abstract and many sentences are overly complicated, especially notations like N/R, F/S, F/R. Could you find simpler, more intuitive ways to present them? Besides, all the experimental results and evidence are hard to follow.

2.In your experiment with the four sample categories, you should consider that different vision models have different training data and training strategies. For example, some detectors are more sensitive to N while others are not. Finer-grained experiments are needed. Also, many of your conclusions have already been reported in plenty of published papers. I don’t see enough novel findings that stand out.

  1. A lot of your experiments only make surface-level observations, without digging deeper to provide supporting evidence.

  2. You also include video experiments in the paper. But I think you haven’t fully analyzed images yet. Why add video experiments at this stage?

Questions for authors

Since SS (self-swap) and FF (cross-identity swap) are generated by the same neural network, they likely share similar generator-specific artifacts. Do you think the low F/SF/S discrimination means detectors are failing to read identity, or are they just grouping both arms together because of shared generation artifacts?

Reviewer_pLZwOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper introduces a four arm protocol built from identical source material, separating detector responses to pipeline processing, neural rerendering, and identity replacement. It reports substantial pipeline false alarms and conditional identity discrimination below the conventional contrast across released checkpoints, and recommends reporting such matched controls alongside standard detection results.

Critical strengths

The paper contributes a source-matched four-arm protocol and evaluates it broadly on released checkpoints across multiple generators, including a held-out generator.

Critical weaknesses
  • No new detection method is proposed and the protocol only recombines established controls, so a conclusion unavailable from the cited precedents is needed.
  • The face swap arms rely on older GAN generators and classic checkpoints, so the contrasts need verification on generators based on diffusion models and recent detectors.
Questions for authors
  • Is recombining existing evaluation controls into a single protocol a sufficient novel contribution?
  • Do the reported contrasts hold on faces generated by current multimodal and diffusion models?
  • The writing is vague and templated, with deeply nested hedged sentences that obscure the argument.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.