DF-PRISM: Beyond Aggregate Metrics via Visual-Attribute Profiling for Deepfake Detectors

Deepfake detection benchmarks typically summarize each detector with aggregate metrics such as AUC over an entire test set, but provide little guidance on which detector to select for a given image. This gap stems from a lack of systematic analysis of how detector performance varies across different image conditions. We operationalize such image conditions as visual attributes, such as illumination, background, occlusion, image stylization, and facial appearance. We present **DF-PRISM** (**D**eep**F**ake **PR**ofiling v**I**a **S**ystematic **M**ulti-attribute Analysis), a pipeline that uses a vision-language model to annotate images with 60 visual attributes and evaluates detectors on attribute-conditioned subsets, yielding a Detector Profile for each detector. The resulting profiles reveal systematic, detector-specific vulnerabilities: for instance, detectors trained primarily on blending artifacts fail to detect cartoon-style fake images. As a proof of concept, we use these profiles to select a subset of detectors for each image based on its annotated attributes; this adaptive selection improves macro-average AUC over an equal-weight ensemble of all 10 detectors across 13 evaluation settings. Overall, DF-PRISM complements aggregate metrics with attribute-conditioned profiling, enabling more informative benchmarking and analysis.