- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DF-PRISM, a visual-attribute profiling framework that characterizes the strengths and weaknesses of deepfake detectors beyond aggregate evaluation metrics. To identify detector-specific vulnerabilities under different visual conditions, DF-PRISM uses a vision-language model to annotate images with 60 binary visual attributes and constructs Detector Profiles through attribute-conditioned AUC and matched-control comparisons.
To demonstrate the utility of these profiles, the paper introduces Adaptive-Ensemble, which selects detectors based on the visual attributes of each test image. The experiments profile 10 pretrained detectors using 5.85M images from ScaleDF and evaluate the approach across 13 benchmark settings. Adaptive-Ensemble improves macro-average AUC from 0.869 to 0.880 compared with an equal-weight ensemble of all detectors.
Critical strengths
-
The proposed framework provides a useful way to understand the behavior of existing deepfake detectors.
Existing benchmarks primarily compare detectors using aggregate metrics, which do not fully characterize their performance under different visual conditions. DF-PRISM addresses this limitation by analyzing detector behavior across 60 visual attributes. For example, Figure 3 shows that cartoon-style images substantially degrade Effort-FF while improving DDA, revealing detector-specific vulnerabilities that aggregate metrics fail to capture.
This provides a more detailed understanding of the strengths and weaknesses of existing detectors, offering useful insights into their behavior under different visual conditions.
-
The proposed Detector Profiles have practical potential for developing more effective detection strategies.
By identifying the visual conditions under which individual detectors perform well or poorly, DF-PRISM provides useful information for designing detection strategies that leverage the complementary strengths of different detectors. The authors demonstrate this potential through Adaptive-Ensemble, which uses attribute-aware detector selection to improve macro-average AUC from 0.869 to 0.880 over Vanilla-Ensemble.
This suggests that Detector Profiles can serve not only as a diagnostic tool but also as a basis for developing practical detection strategies, such as adaptive detector selection and ensemble methods.
Critical weaknesses
-
The selection and characterization of the evaluated detectors are insufficient to support the generality of the proposed framework.
The paper should evaluate a broader and more representative set of recent deepfake detectors, particularly those designed to improve generalization and robustness across diverse visual conditions. However, five of the ten evaluated detectors rely on generic vision architectures, despite the availability of numerous deepfake-specific approaches developed with diverse forensic cues and learning strategies. It remains unclear whether the proposed framework would remain effective with a broader set of recent generalization-oriented detectors.
Furthermore, the paper lacks a systematic analysis of how detector characteristics relate to attribute sensitivity. Different training data (e.g., facial manipulation datasets [1, 2] versus general AI-generated image datasets [3, 4]) and detection strategies (e.g., spatial-based, frequency-based, CLIP-based, and multimodal approaches, etc.) may lead to different vulnerability patterns. Analyzing these factors would better explain the observed detector profiles and strengthen the generality of the proposed framework.
-
The practical utility of the proposed Detector Profiles could be further explored.
I believe that a key contribution of this paper is its ability to characterize the strengths and weaknesses of existing detectors, which could also help develop more effective detection strategies. While the authors demonstrate this potential through Adaptive-Ensemble, the evaluation relies solely on AUC, which does not necessarily reflect binary classification performance at a fixed decision threshold. Evaluating the proposed approach on more challenging and real-world benchmarks [5–7], and examining whether it improves detection accuracy under a fixed threshold, would further strengthen the practical implications of this work.
References
[0] Yan et al., "DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection," NeurIPS, 2023.
[1] Yan, Zhiyuan, et al. "Df40: Toward next-generation deepfake detection." Advances in Neural Information Processing Systems 37 (2024): 29387-29434.
[2] Rössler et al., "FaceForensics++: Learning to Detect Manipulated Facial Images," ICCV, 2019.
[3] Wang et al., "CNN-Generated Images Are Surprisingly Easy to Spot... for Now," CVPR, 2020.
[4] Zhu et al., "GenImage: A Million-Scale Benchmark for Detecting AI-Generated Image," NeurIPS, 2023.
[5] Yan et al., "A Sanity Check for AI-Generated Image Detection," ICLR, 2025.
[6] Chandra et al., "Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circulated in 2024," arXiv:2503.02857, 2025.
[7] Lin et al., "Fit for Purpose? Deepfake Detection in the Real World," arXiv:2510.16556, 2025.
Questions for authors
- Does the improvement in AUC translate into better detection accuracy on real-world datasets?
As mentioned in Weakness 2, AUC does not necessarily reflect binary classification performance at a fixed threshold. I would be interested in whether Adaptive-Ensemble also improves detection accuracy over Vanilla-Ensemble using a fixed threshold of 0.5 on real-world benchmarks [5–7].
- Does the proposed approach remain effective when using only recent generalization-oriented deepfake detectors?
Given that several evaluated detectors rely on generic vision architectures, I would like to know whether the benefits of attribute-aware ensemble selection persist when the detector pool consists of more recent deepfake-specific approaches.