acceptodds
Under review as a conference paper at ICLR 2027

DF-PRISM: Beyond Aggregate Metrics via Visual-Attribute Profiling for Deepfake Detectors

Abstract

Deepfake detection benchmarks typically summarize each detector with aggregate metrics such as AUC over an entire test set, but provide little guidance on which detector to select for a given image. This gap stems from a lack of systematic analysis of how detector performance varies across different image conditions. We operationalize such image conditions as visual attributes, such as illumination, background, occlusion, image stylization, and facial appearance. We present **DF-PRISM** (**D**eep**F**ake **PR**ofiling v**I**a **S**ystematic **M**ulti-attribute Analysis), a pipeline that uses a vision-language model to annotate images with 60 visual attributes and evaluates detectors on attribute-conditioned subsets, yielding a Detector Profile for each detector. The resulting profiles reveal systematic, detector-specific vulnerabilities: for instance, detectors trained primarily on blending artifacts fail to detect cartoon-style fake images. As a proof of concept, we use these profiles to select a subset of detectors for each image based on its annotated attributes; this adaptive selection improves macro-average AUC over an equal-weight ensemble of all 10 detectors across 13 evaluation settings. Overall, DF-PRISM complements aggregate metrics with attribute-conditioned profiling, enabling more informative benchmarking and analysis.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Deepfake Detection, Model Evaluation, Benchmark Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fxCYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DF-PRISM, a visual-attribute profiling framework that characterizes the strengths and weaknesses of deepfake detectors beyond aggregate evaluation metrics. To identify detector-specific vulnerabilities under different visual conditions, DF-PRISM uses a vision-language model to annotate images with 60 binary visual attributes and constructs Detector Profiles through attribute-conditioned AUC and matched-control comparisons.

To demonstrate the utility of these profiles, the paper introduces Adaptive-Ensemble, which selects detectors based on the visual attributes of each test image. The experiments profile 10 pretrained detectors using 5.85M images from ScaleDF and evaluate the approach across 13 benchmark settings. Adaptive-Ensemble improves macro-average AUC from 0.869 to 0.880 compared with an equal-weight ensemble of all detectors.

Critical strengths
  1. The proposed framework provides a useful way to understand the behavior of existing deepfake detectors.

    Existing benchmarks primarily compare detectors using aggregate metrics, which do not fully characterize their performance under different visual conditions. DF-PRISM addresses this limitation by analyzing detector behavior across 60 visual attributes. For example, Figure 3 shows that cartoon-style images substantially degrade Effort-FF while improving DDA, revealing detector-specific vulnerabilities that aggregate metrics fail to capture.

    This provides a more detailed understanding of the strengths and weaknesses of existing detectors, offering useful insights into their behavior under different visual conditions.


  1. The proposed Detector Profiles have practical potential for developing more effective detection strategies.

    By identifying the visual conditions under which individual detectors perform well or poorly, DF-PRISM provides useful information for designing detection strategies that leverage the complementary strengths of different detectors. The authors demonstrate this potential through Adaptive-Ensemble, which uses attribute-aware detector selection to improve macro-average AUC from 0.869 to 0.880 over Vanilla-Ensemble.

    This suggests that Detector Profiles can serve not only as a diagnostic tool but also as a basis for developing practical detection strategies, such as adaptive detector selection and ensemble methods.

Critical weaknesses
  1. The selection and characterization of the evaluated detectors are insufficient to support the generality of the proposed framework.

    The paper should evaluate a broader and more representative set of recent deepfake detectors, particularly those designed to improve generalization and robustness across diverse visual conditions. However, five of the ten evaluated detectors rely on generic vision architectures, despite the availability of numerous deepfake-specific approaches developed with diverse forensic cues and learning strategies. It remains unclear whether the proposed framework would remain effective with a broader set of recent generalization-oriented detectors.

    Furthermore, the paper lacks a systematic analysis of how detector characteristics relate to attribute sensitivity. Different training data (e.g., facial manipulation datasets [1, 2] versus general AI-generated image datasets [3, 4]) and detection strategies (e.g., spatial-based, frequency-based, CLIP-based, and multimodal approaches, etc.) may lead to different vulnerability patterns. Analyzing these factors would better explain the observed detector profiles and strengthen the generality of the proposed framework.


  1. The practical utility of the proposed Detector Profiles could be further explored.

    I believe that a key contribution of this paper is its ability to characterize the strengths and weaknesses of existing detectors, which could also help develop more effective detection strategies. While the authors demonstrate this potential through Adaptive-Ensemble, the evaluation relies solely on AUC, which does not necessarily reflect binary classification performance at a fixed decision threshold. Evaluating the proposed approach on more challenging and real-world benchmarks [5–7], and examining whether it improves detection accuracy under a fixed threshold, would further strengthen the practical implications of this work.


References

[0] Yan et al., "DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection," NeurIPS, 2023.

[1] Yan, Zhiyuan, et al. "Df40: Toward next-generation deepfake detection." Advances in Neural Information Processing Systems 37 (2024): 29387-29434.

[2] Rössler et al., "FaceForensics++: Learning to Detect Manipulated Facial Images," ICCV, 2019.

[3] Wang et al., "CNN-Generated Images Are Surprisingly Easy to Spot... for Now," CVPR, 2020.

[4] Zhu et al., "GenImage: A Million-Scale Benchmark for Detecting AI-Generated Image," NeurIPS, 2023.

[5] Yan et al., "A Sanity Check for AI-Generated Image Detection," ICLR, 2025.

[6] Chandra et al., "Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circulated in 2024," arXiv:2503.02857, 2025.

[7] Lin et al., "Fit for Purpose? Deepfake Detection in the Real World," arXiv:2510.16556, 2025.

Questions for authors
  1. Does the improvement in AUC translate into better detection accuracy on real-world datasets?
    As mentioned in Weakness 2, AUC does not necessarily reflect binary classification performance at a fixed threshold. I would be interested in whether Adaptive-Ensemble also improves detection accuracy over Vanilla-Ensemble using a fixed threshold of 0.5 on real-world benchmarks [5–7].

  1. Does the proposed approach remain effective when using only recent generalization-oriented deepfake detectors?
    Given that several evaluated detectors rely on generic vision architectures, I would like to know whether the benefits of attribute-aware ensemble selection persist when the detector pool consists of more recent deepfake-specific approaches.
Reviewer_FQs9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DF-PRISM, an analysis pipeline that uses a vision-language model (VLM) to annotate images with 60 observable visual attributes and, on that basis, builds attribute-conditioned "Detector Profiles" for deepfake detectors, revealing detector-specific weaknesses that aggregate AUC conceals.

Critical strengths
  1. A novel and practical framing: evaluation is sliced by observable visual attributes instead of unobservable forgery types. Existing benchmarks (DeepfakeBench, DF40, ScaleDF) slice by dataset or forgery method (face swap, reenactment, diffusion generation), but forgery type is unknown at deployment. The authors condition instead on directly observable attributes such as lighting, occlusion, stylization and skin texture, so profiles are usable at test time. This is a real increment over fairness studies (which cover only race, gender and skin tone) and over generic ensemble work. The motivating experiment in Figure 1 also concisely supports "aggregate metric ≠ per-image optimum": Effort, the lowest-AUC detector, still gives the best-calibrated prediction on 30.4% of images.

  2. The experimental scale and coverage are solid for an analysis paper. Offline profiling covers 5.85M images × 60 attributes × 10 heterogeneous detectors (Xception, ConvNeXt, Swin, SegFormer, ForensicsAdapter, DDA, CDFA, D3, Effort-FF). Evaluation spans 13 settings (7 established benchmarks plus 6 HydraFake cross-domain generators), at about 1,800 GPU-hours, with a per-stage breakdown in Appendix J. The detectors vary in backbone, training data and training paradigm, so the conclusions are unlikely to be quirks of a single model.

  3. The matched control is more rigorous than a direct with/without-attribute comparison. AUC_no is not a whole-dataset baseline. It comes from a control group of nearest neighbors by Hamming distance over the other K−1 attributes, paired with the same real/fake label (Sec 3.2). This curbs confounding from attribute co-occurrence with forgery type or identity, so ΔAUC is closer to the effect of the attribute itself. Together with three sensitivity thresholds (±0.05) and the full 10×60 heatmaps in Appendix D, the analysis is easy to verify.

Critical weaknesses
  1. The core method, Adaptive-Ensemble, is too simple, relies on hard exclusion by relative ranking, and lacks baseline comparisons. The rule is just: for each attribute, exclude the B=3 detectors with the lowest AUC_yes, take the union over active attributes, and average the rest with equal weights. There are two problems. Exclusion is relative to the profile. Even if all detectors do well on an attribute, three are still dropped. The CDFA incident in Appendix F is a direct result of this flaw, yet the authors offer only a post-hoc diagnosis and no fix, such as absolute-threshold exclusion, ΔAUC weighting instead of hard exclusion, keeping the top-k, or normalizing by AUC_base. There is only one comparison baseline, Vanilla-Ensemble. There is no comparison with weighted ensembles, learned meta-routers, confidence/uncertainty-based selection (e.g., max-softmax or entropy routing), or a simple per-benchmark oracle selection. It is therefore impossible to tell whether profiles are more useful than any per-image routing signal. The learned fusion networks mentioned in related work should have been required baselines.

  2. The gain is small, there is no significance testing, and B=3 appears to have been picked on the evaluation sets. The +1.1% macro-average gain (0.869→0.880) gets worse on 2 of 13 settings, yet there are no variances, confidence intervals, repeated runs or significance tests (e.g., DeLong for AUC). The B ablation in Appendix G shows a mean of 0.869 at B=1 (equal to the equal-weight ensemble) and 0.867 at B=5 (worse than it). Only B=3 reaches 0.880, and it happens to be optimal on the 13 final evaluation settings, which risks hyperparameter selection on the test set. There is no independent dev/test split and no cross-dataset stability analysis of B (e.g., choosing B on the 7 benchmarks and validating on HydraFake).

  3. Overlap between the profiling source and the evaluation sets, plus training-set-dependent score calibration, creates potential leakage and circularity. Profiles come entirely from the ScaleDF train split, while the 13 evaluation settings include the ScaleDF test split. Detector scores are also mapped to [0,1] using empirical quantiles on the ScaleDF training set. The gain in the ScaleDF column (0.847→0.868) is therefore partly in-distribution self-evaluation and is less convincing. Whether profiles still hold under domain shift is discussed only through the two failure cases (CDFv2, ForgeryNet). There is no systematic transferability analysis, such as ordering datasets by domain distance from ScaleDF to see how gains decay. Nor is there a cross-source ablation (profile on dataset A, evaluate on dataset B), which is the experiment this kind of work most needs.

  4. The evidence for the attribute taxonomy and annotation reliability is insufficient to support causal conclusions. The 60 attributes are hand-designed (the authors admit they are a diagnostic starting point, not an exhaustive description) and all binarized, discarding intensity (there is no degree to "extreme backlighting"). Attribute co-occurrence and interaction are not modeled. Taking the union over active attributes implicitly assumes the exclusion lists are independent and stackable, whereas "black-and-white + cartoon + compression" are highly correlated in practice. The human audit covers only 200 images with 2 annotators and reports no inter-annotator agreement (e.g., Cohen's κ). The authors note that sparse positives dilute the overall agreement rate, so recall and precision on the positive class are the key metrics, yet they are not given. When the matched control's ΔAUC is used for explanatory attribution, imperfect pairing may leave residual confounding. The paper glosses over this with the vague phrase "differing as much as possible only in the target attribute" and does not quantify match quality, such as post-matching covariate balance.

Questions for authors
Reviewer_cxs8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

DF-PRISM characterizes deepfake detectors through attribute-conditioned evaluation. The authors use Qwen3.5-9B to annotate 60 visual attributes on 5.85 million ScaleDF images and construct profiles for 10 pretrained detectors. These profiles summarize conditional AUCs and performance differences relative to matched attribute-absent controls. As a downstream validation, an attribute-guided ensemble excludes poorly ranked detectors for each input, improving macro-average AUC from 0.869 to 0.880 across 13 evaluation settings.

Critical strengths
  1. The attribute-conditioned analysis reveals meaningful detector complementarity that aggregate scores obscure. For example, on cartoon-style images, DDA achieves an AUC of 0.948, whereas Effort-FF achieves 0.564; their matched contrasts are +0.064 and −0.225, respectively (Appendix H). Such opposing responses under the same condition provide useful information for detector evaluation and auditing.

  2. The study combines substantial profiling coverage with evaluation across multiple domains. The downstream improvement from 0.869 to 0.880 is encouraging, with larger gains on DFF and GPT-4o. This provides preliminary evidence that the profiles may support practical decisions beyond descriptive visualization.

Critical weaknesses
  1. The downstream evaluation does not isolate the benefit of image-specific attributes from the benefit of removing globally weak detectors. The principal ensemble baseline averages all 10 detectors. A stronger, directly relevant control would use the same source-domain data to select a fixed detector subset and apply it to every test image.

    Appendix F makes this concern concrete: CDFA ranks in the bottom three on all 50 routing attributes and ranks last on 36. Consequently, every image activating at least one routing attribute excludes CDFA, regardless of which attribute is present. This suggests that part of the improvement could come from systematic detector pruning rather than matching detectors to individual images. The current experiments do not distinguish these explanations.

    Although the router is presented as a proof of concept, it is the main experimental validation of the profiles’ actionable value. A comparison against attribute-independent subset selection is therefore important for interpreting this contribution. This concern does not invalidate the reported conditional performance differences.

  2. The annotation audit does not sufficiently establish the reliability of the positive attribute subsets underlying the main diagnostic findings. The human audit covers 200 class-balanced images and primarily reports overall agreement across attribute judgments. The authors appropriately acknowledge that rare positive labels affect the interpretation of these agreement rates. However, conditional AUCs and routing decisions depend specifically on which images are labeled attribute-positive.

    This matters for headline findings such as CDFA’s −0.212 matched contrast for face masks (Table 8), an attribute appearing toward the low-frequency end of Figure 4. The paper does not report the human-audited positive support or positive-label accuracy for these key attributes. Cross-VLM agreement provides useful consistency evidence, but it does not establish agreement with human judgments on the relevant positive subsets. The semantic reliability of several consequential profile entries therefore remains insufficiently validated.

Questions for authors
  1. Could the authors compare Adaptive-Ensemble against a fixed-subset ensemble selected using only ScaleDF source-domain data? A minimal implementation would rank detectors by their overall source-domain AUC and evaluate fixed subsets with sizes comparable to those typically retained by Adaptive-Ensemble. Please use the same score normalization and report results on the same 13 settings, without selecting the subset using target-domain labels. This would directly test whether image-specific attributes add value beyond source-domain detector ranking.

  2. Could the authors provide a targeted human audit for a small number of attributes supporting the main conclusions, particularly face masks and cartoon style? Please report the audited positive counts separately for real and fake images, assess the accuracy of Qwen-positive labels and the corresponding attribute-absent controls, and check whether the key detector rankings and matched AUC differences remain on the manually verified subsets. This would address the reliability of the central findings without requiring manual reannotation of all 60 attributes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.