acceptodds
Withdrawn from ICLR 2027

Fake-in-Facext: Towards Fine-Grained Explainable DeepFake Analysis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The advancement of Vision-Language Models (VLMs) has bridged the gap between vision and language tasks, enabling the implementation of Explainable DeepFake Analysis (XDFA). However, current methods suffer from a lack of fine-grained awareness: the description of artifacts in data annotation is unreliable and coarse-grained, and the models fail to support the output of connections between textual forgery explanations and the visual evidence of artifacts, as well as the input of queries for arbitrary facial regions. As a result, these VLMs can not fully leverage prior knowledge of Face Context (Facext) to enhance performance and broaden application scope. To address this limitation, we propose the Fake-in-Facext (FiFa) framework, with contributions focusing on data annotation and model construction. We first define a Facial Image Concept Tree (FICT) to divide facial images into fine-grained regional concepts, thereby obtaining a more reliable data annotation pipeline, FiFa-Annotator, for forgery explanation. We validate that the data generated by FiFa-Annotator can effectively improve detection generalization. Based on this dedicated data annotation, we introduce a novel Artifact-Grounding Explanation (AGE) task, which generates textual forgery explanations interleaved with segmentation masks of manipulated artifacts. We propose a unified multi-task learning architecture, FiFa-VLM, to simultaneously support abundant multimodal inputs and outputs for fine-grained Explainable DeepFake Analysis. With multiple auxiliary supervision tasks, FiFa-VLM can outperform strong baselines on the AGE task and achieve SOTA performance on existing XDFA datasets. The code and data will be made open-source.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Explainable DeepFake Analysis, Vision-Language Model, Face Context

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_V9CSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Fake-in-Facext, a framework for fine-grained explainable face-forgery analysis. It uses a hierarchical facial concept tree to generate region-specific explanations and artifact masks, producing 1.38 million QA pairs. FiFa-VLM combines a face-pretrained encoder, box prompts, and a multi-task decoder to support detection, localization, and explanations interleaved with segmentation masks.

Critical strengths
  1. Linking individual explanation phrases to masks is a useful extension beyond image-level explanations. Region and box queries also provide a practical way to inspect specific facial areas.

  2. The facial concept hierarchy gives the annotations a consistent spatial vocabulary. The resulting dataset supports several related tasks within one framework.

  3. The experiments include annotation-source comparisons, architectural ablations, and a trainable-vision GLaMM baseline. These controls provide evidence that the proposed supervision and face-specific representation improve performance on FiFa-Bench.

Critical weaknesses
  1. The artifact ground truth is not sufficiently justified. Section 3.2 defines artifact masks using the top 5% of pixel differences between real and manipulated images. This identifies large changes, but does not establish that they are visible forensic artifacts. It may miss subtle defects or include differences caused by alignment and compression. The fixed percentile and concept-selection thresholds are not validated against independent human annotations. The treatment of fully synthesized faces, which lack a corresponding original image, also needs clarification.

  2. The evidence for reduced hallucination is partly circular. Table 5 evaluates concept precision and recall against labels produced by the same rules used to construct the training annotations. This tests agreement with the annotation pipeline, not whether claims about lighting, texture, or geometry are correct. Likewise, METEOR and mask IoU are reported separately, leaving the central phrase-to-evidence connection insufficiently evaluated. A model could identify the right facial region while giving an incorrect explanation.

  3. The annotation prompt can encourage unsupported explanations. Appendix A.3 requires every selected concept to be discussed as a tampered region, without an explicit option to report insufficient evidence. It also lists gender-associated hairstyles and makeup as possible forgery cues. These characteristics are not inherently evidence of manipulation and could introduce systematic bias. More precise region names do not, by themselves, make the accompanying claims more reliable.

  4. The dataset’s effective diversity and split independence are unclear. The reported 1.38 million examples are QA pairs, with many region and box queries derived from the same images. Table 16 reports task-level counts but does not establish the number of unique images or whether all related queries, frames, and identities remain within one split. In addition, augmented boxes are retained only when they contain at least three artifact-existing concepts. This leaves uncertainty about performance on clean regions, very small edits, and arbitrary user-selected boxes.

  5. Generalization claims are stronger than the supporting results. Table 6 evaluates FiFa-Annotator+CLIP rather than FiFa-VLM’s grounded explanations. Its AUC gains over FFTG+CLIP are modest: 0.23, 0.44, and 0.55 percentage points on the three cross-domain datasets, without uncertainty estimates. Table 8 similarly reduces cross-domain ACER from 29.32% to 28.57%. These results suggest some benefit, but do not establish robust transfer of explanation and localization capabilities.

  6. The architectural contribution is not fully separated from face-specific pretraining and additional supervision. Table 7 includes a useful trainable GLaMM control, but replacing its visual encoders also changes the pretraining domain and encoder structure. The single-task comparison matches task-specific sample counts, while the multi-task model receives additional supervision from other tasks. This supports the usefulness of joint training, but does not isolate architectural gains or establish an advantage under equal total training budgets.

Questions for authors
  • How accurately do the difference-based masks match independently annotated visible artifacts? How sensitive are the results to the 5% threshold and the concept-selection thresholds?

  • How are artifact masks constructed for fully synthesized faces without paired originals? What alignment and compression controls are applied before computing image differences?

  • Can you provide a blinded human audit of the explanations, checking whether each claimed defect is actually visible? Does allowing the annotator to report insufficient evidence reduce hallucinations?

  • How are generated phrases matched to ground-truth masks during AGE evaluation? Can you measure whether both the region and the associated artifact description are correct?

  • How many unique images and identities underlie the QA pairs, and are all related examples kept within the same split? How does the model perform on boxes containing no artifacts or only one small artifact?

  • Can FiFa-VLM retain its explanation and localization performance on unseen manipulation methods without retraining? Please distinguish these results from the CLIP-based annotation-transfer experiment and report variation across training runs.

Reviewer_xCrKOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FiFa uses a Facial Image Concept Tree (FICT), paired-image RGB differences, landmarks, and face parsing to produce regional masks, with GPT-4o/ChatGPT generating explanations. The authors construct FiFa-Instruct-1M with approximately 1.38M QA-pairs, FiFa-Bench, and the FiFa-11 task suite. FiFa-VLM combines a single FaRL-ViT-B, a Box Encoder, and a Multi-Task Decoder to support Artifact-Grounding Explanation (AGE) and region-specific queries. Evaluation covers FiFa-Bench, DD-VQA, DFA-Bench, cross-dataset detection, and ablations.

Critical strengths
  1. FICT’s 112 Atomic Concepts provide more specific locations than descriptions such as “around the eyes.” Constraining annotation with spatial priors and linking text to masks through AGE provide a useful design that supports Region-Level and Box-Level queries.

  2. Replacing two visual encoders with one FaRL-ViT-B reportedly saves 0.94B parameters. On FiFa-Bench, ACER decreases from 6.97 to 4.47 and I-AGE mIoU increases from 22.5 to 29.6. Table 7 supports Region Mask auxiliary supervision, while external benchmarks and cross-dataset detection provide additional evidence.

Critical weaknesses
  1. Section 3.2 derives Artifact Masks from the Top 5% of paired-image RGB differences, but pixel change is not equivalent to visible forgery evidence and may reflect compression, color, or registration changes. Landmark and face-parsing errors could also propagate into explanations, without being quantified. FiFa-Bench uses labels from the same pipeline, and Table 5 uses Step 2 outputs as ground truth for concept precision/recall. These mainly measure agreement with automated rules. External detection and text metrics do not replace independent human annotation of textual factuality and visible evidence localization.

  2. The Multi-Task Decoder, task-specific queries, and Region Mask auxiliary supervision resemble existing multi-task segmentation designs. Table 7 includes useful ablations, but face pretraining, encoder architecture, and parameter count change together, leaving the single-encoder design’s contribution partly unresolved. Comparisons against stronger VLMs on explanation tasks are limited; GPT-4o in Table 5 only compares annotation sources. DFA-Bench gains are small: 92.74→92.95 in classification accuracy and 42.54→42.78 in ROUGE-L. These results support system effectiveness but are less convincing about the model design’s incremental contribution and broader superiority.

Questions for authors
  1. Is there a blinded human evaluation independent of FiFa-Annotator labels that assesses textual factuality and visible forgery evidence localization? Please provide the annotation protocol, inter-annotator agreement, and results.

  2. How much do the Top-5% RGB difference masks overlap with human-identified forgery evidence? How do thresholds, compression, registration, and landmark/face-parsing errors affect annotations?

  3. Can the effects of face pretraining and the single-encoder design be separated under the same data and training budget, with parameter counts matched as closely as possible?

Reviewer_mNeHOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Fake-in-Facext (FiFa), a framework for fine-grained explainable deepfake analysis that connects textual forgery explanations with localized visual evidence. To improve forgery annotations, the authors introduce a Facial Image Concept Tree (FICT), which divides facial images into 112 atomic concepts and 72 parent concepts. Based on this hierarchy, FiFa-Annotator generates region-specific forgery explanations and artifact masks, forming the FiFa-Instruct-1M training dataset and FiFa-Bench evaluation benchmark.

The paper further introduces the Artifact-Grounding Explanation (AGE) task and proposes FiFa-VLM, a unified multi-task architecture supporting detection, manipulation classification, localization, and artifact-grounded explanations. The experiments show improvements over GLaMM on FiFa-Bench and competitive results on DD-VQA and DFA-Bench. Ablation studies also demonstrate the benefits of multi-task supervision for detection and localization.

Critical strengths
  1. The paper provides a useful framework for fine-grained and visually grounded deepfake explanations.

    Existing explainable deepfake detection methods typically provide textual explanations or localization results without explicitly connecting them. FiFa addresses this limitation by introducing the AGE task, which associates textual descriptions of forgery artifacts with corresponding segmentation masks. The Facial Image Concept Tree further supports more detailed descriptions of individual facial regions.


  1. The proposed framework integrates diverse deepfake analysis tasks within a unified architecture.

    FiFa-VLM supports detection, manipulation classification, localization, and explanation tasks using a shared face-pretrained vision encoder. The experiments show improvements over the fine-tuned GLaMM baseline on FiFa-Bench, particularly in localization and artifact grounding. Table 7 also shows that auxiliary supervision contributes to performance improvements.

Critical weaknesses
  1. The baseline comparisons with existing explainable deepfake detection methods are insufficient.

    The evaluations in Tables 2–4 rely on limited baselines. Table 2 compares FiFa-VLM only with GLaMM, a general-purpose pixel-grounding model, while Tables 3 and 4 each include only one existing XDFA method. Although Table 1 introduces several relevant approaches, including FFAA and FakeShield, these models are not included in the performance comparisons, even for their supported tasks.

    Furthermore, the localization and artifact-grounding performance is evaluated only on the authors' FiFa-Bench, without validation on existing external localization benchmarks. More comprehensive comparisons with existing XDFA methods, particularly those supporting forgery localization, would provide stronger evidence of FiFa-VLM's effectiveness and generalization.


  1. The cross-domain detection capability of FiFa-VLM is not sufficiently validated.

    The authors present FiFa-VLM as a unified framework supporting detection, classification, localization, and explanation. However, Tables 6 and 18 evaluate cross-domain detection using a separately trained CLIP detector rather than FiFa-VLM itself. While these experiments validate the utility of FiFa-Annotator, they do not establish the detection generalization of FiFa-VLM. Although Table 8 reports cross-domain detection results, it does not compare FiFa-VLM with existing detectors.

    Furthermore, Tables 6 and 18 provide limited comparisons with recent state-of-the-art vision-based deepfake detectors. I believe that direct cross-domain comparisons with recent vision-based and MLLM-based deepfake detectors would provide stronger evidence of FiFa-VLM's detection capability and generalization.

Questions for authors
  1. How are real images handled in the explanation and AGE tasks?

    The annotation pipeline mainly describes forgery artifacts in manipulated images. Do the explanation and AGE training sets also contain real images or regions without forgery artifacts? If not, how does FiFa-VLM learn to recognize the absence of artifacts rather than generating potentially hallucinated explanations?


  1. How does FiFa-Annotator handle entirely synthesized face images?

    The annotation pipeline identifies artifact regions through pixel-level differences between real and manipulated images. However, entirely synthesized faces do not necessarily have corresponding real images. How are artifact masks and explanations generated for these cases? Are they excluded from the annotation pipeline, or is a different procedure used?


  1. How is the reliability of GPT-generated forensic annotations validated?

    FiFa-Annotator relies on GPT-4o to generate artifact descriptions from predefined regions, which may introduce systematic bias or hallucinated explanations. Previous studies have adopted multi-MLLM annotation and consensus strategies to mitigate such issues (e.g., Veritas [1] and FakeVLM [2]). Does FiFa-Annotator include any independent verification of its generated explanations? Since Table 5 evaluates concept precision and recall using labels derived from the same annotation pipeline, I believe that additional validation of annotation quality and hallucination would strengthen the paper.


  1. What additional benefits does AGE provide over conventional artifact localization?

    I believe that localizing forgery artifacts already provides substantial interpretability by highlighting suspicious regions. Therefore, the additional value of connecting these regions with textual explanations is not entirely clear. Although Tables 7 and 8 demonstrate the benefits of multi-task supervision, they do not directly isolate the contribution of jointly grounded explanations. Is there an experiment showing that AGE provides additional benefits over performing localization and textual explanation separately?


References

[1] Tan et al., "Veritas: Generalizable Deepfake Detection via Pattern-Aware Reasoning," ICLR, 2026.

[2] Wen et al., "Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection with Artifact Explanation," NeurIPS, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.