- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how vision-language models (VLMs) can use forensic evidence to make decisions in face forgery detection. The authors propose PDM-Face, which maps features extracted by forensic experts into continuous evidence tokens and feeds them into a VLM alongside the image. The model answers eight forensic hypotheses in parallel, then combines the answer probabilities with the expert scores to produce a final forgery probability.
Critical strengths
Considering generation mechanisms, spectral characteristics, and expert outputs as complementary sources of evidence and integrating them into a joint decision is a sensible design choice.
Critical weaknesses
-
The need for the VLM decision layer remains unclear. The current baseline averages the five CNN predictions before combining them with CLIP. A simple router using CLIP features to dynamically weight individual experts could be more competitive. Without this comparison, it is unclear whether the VLM provides enough additional benefit to justify its computational cost.
-
The performance drops and expert selection need better justification. PDM-Face performs worse than the expert-only baseline on 3rdBench and WildDeepfake, but the causes are not sufficiently explained. The rationale for selecting the five CNNs and CLIP is also unclear. These gaps make it difficult to assess the method’s reliability across different datasets and expert configurations.
Questions for authors
-
Could you compare PDM-Face with a lightweight router that uses CLIP image features to dynamically weight individual expert predictions, rather than averaging the CNN predictions first? Please use the same experts and training data, control for access to evidence features and auxiliary supervision, and report both AUROC and end-to-end inference cost. This would help establish the additional value of the VLM decision layer.
-
What criteria guided the selection of the five CNNs and the CLIP classifier, and how complementary are they? Why average the CNN predictions before gating and require the CNN ensemble to receive at least 50% of the weight?
-
Why does PDM-Face underperform the expert-only baseline on 3rdBench and WildDeepfake? Are these drops consistent across seeds and statistically significant? Please analyze when the decision layer worsens expert predictions and whether the hypothesis branches or final fusion contribute to these errors.
3 comments
Thanks for the reviewer's precise and constructive suggestions (Part 2/2)
W2 / Q2: Selection criterion. The two experts were chosen for complementary strengths. The CNN ensemble is a low-level forensic expert. It consists of five four-branch networks with an EfficientNetV2-S backbone, and each network fuses RGB, spectral, residual, and self-blending streams. The logistic CLIP ViT-L/14 probe is a semantic expert. A gate reads the CLIP features and weights the two.
Complementarity. Each expert leads on different benchmarks (AUROC):
| Expert | In-domain | 3rdBench | GenImage | FF++ | DFDC | WildDeepfake |
|---|---|---|---|---|---|---|
| CNN ensemble | 0.996 | 0.964 | 0.741 | 0.575 | 0.571 | 0.724 |
| CLIP probe | 0.957 | 0.840 | 0.854 | 0.660 | 0.629 | 0.636 |
The Spearman rank correlation between the scores of the two experts on the external benchmarks is 0.15 to 0.56. The two experts therefore make different errors, which is the condition under which gating helps.
Why the five CNNs are averaged before gating. The five CNNs share the same four-branch architecture, so they read the same kind of evidence. Averaging them, with flip test-time augmentation and per-model temperatures, is standard ensembling that reduces variance before gating. The gate then weighs two different kinds of evidence, forensic and semantic.
Why the 50% floor. The CNN ensemble is the stronger in-domain detector (0.996 against 0.957). The gate reads CLIP features, and these features shift on unseen sources. The floor keeps the forensic CNN in every decision when that happens.
We will add the per-expert AUROC table and a gate-floor ablation to the appendix.
W2 / Q3: Consistency and significance. On 3rdBench the difference is consistent and small. PDM-Face minus the expert layer is -0.011, -0.013, and -0.012 for the three seeds, and it is significant (95% CI [-0.013, -0.008] for seed 0). On WildDeepfake the difference is not consistent. It is -0.005 and -0.002 for two seeds (not significant) and -0.026 for the third. On the full WildDeepfake test set the two are statistically tied (0.742 against 0.746, 95% CI [-0.009, +0.002]).
When the decision layer changes the expert ranking. We ran a new per-image analysis on 3rdBench.
- By source: on the text-to-image subset both reach 1.000. The small difference comes from the StyleGAN set and the two face-swap collections.
- By expert confidence: 76% to 85% of the difference comes from the half of the images where the expert layer is least confident. On the images where the expert layer is confident, PDM-Face leaves the ranking essentially unchanged.
The decision layer therefore acts mostly where the experts are uncertain. This is where it gains on FF++ and DFDC (+0.034 and +0.021 on the full test sets), which contain compressed video, face swaps, and reenactment. It is also where it costs about 0.01 on 3rdBench, where the expert layer is already at 0.977 and has little room to improve.
Branches or fusion. The difference does not originate in the hypothesis branches or in the learned fusion. A decision layer with a single question and no fusion is lower on 3rdBench (0.948). The eight parallel hypotheses raise this to 0.965. Replacing the learned fusion with a plain average of the hypothesis answers lowers 3rdBench by 0.015. Both components therefore move the decision toward the expert layer. The remaining difference comes from the re-ranking of low-confidence images described above.
We will add the per-seed intervals and the per-source and per-confidence analysis to the appendix, and we will report both findings in the main text.
We thank the reviewer again. The proposed router experiment and the request for a failure analysis gave the paper two results that it lacked, and both will appear in the revision.
Additional results on the expert layer: gate floor and the complete fusion table
This note adds to our earlier response. The gate-floor ablation that we promised there is finished, and the fusion comparison is now run at every seed, which updates the single-seed test given there. Both are in the revised PDF.
Gate floor. The gate gives the CNN ensemble the weight g = c + (1 - c) * sigmoid(MLP(CLIP feature)), so c is the minimum weight of the CNN ensemble. We retrained the gate with c = 0 (no floor), c = 0.25, and c = 0.5 on the 60,000-face training pool, with three seeds each, and evaluated the expert layer alone.
| Gate (AUROC, mean of 3 seeds) | Celeb-DF | 3rdBench | GenImage | DFDC | FF++ | WildDeepfake | Mean |
|---|---|---|---|---|---|---|---|
| c = 0 (no floor) | 0.838 | 0.968 | 0.765 | 0.587 | 0.587 | 0.725 | 0.745 |
| c = 0.25 | 0.857 | 0.971 | 0.798 | 0.589 | 0.595 | 0.734 | 0.757 |
| c = 0.5 | 0.854 | 0.972 | 0.795 | 0.595 | 0.597 | 0.737 | 0.758 |
With either floor, the three-seed mean is at least as high as without a floor on every external benchmark, and the six-benchmark mean rises from 0.745 to 0.757 and 0.758. In the paired test at seed 0 (last rows of Table 11), the gate with c = 0.5 is significantly higher than the gate without a floor on Celeb-DF, 3rdBench, GenImage, and WildDeepfake (by +0.005 to +0.027), and the two are tied on DFDC and FF++. These gates are a re-implementation of the gate, trained with the recipe of the light fusion heads.
Complementarity within each class. The rank correlation in our earlier response was computed over all images, so it also contains the agreement of the two experts on the label itself. Among real faces alone and among forged faces alone, the Spearman correlation between the two expert scores is 0.01 to 0.36 on the six external benchmarks, so their evidence is largely non-redundant. Each expert alone also leads on different benchmarks: the CNN ensemble on Celeb-DF (0.838 against 0.794), 3rdBench, and WildDeepfake, and the CLIP probe on GenImage, DFDC, and FF++. Each of these six differences is significant in a paired test.
Fusion table. Table 3 of the revision now gives all four heads on all six benchmarks as three-seed means, with a paired test at every seed: against the selected head (the multi-task MLP of our first response), PDM-Face is higher on DFDC, FF++, and WildDeepfake (significant at two of three seeds each) and lower on Celeb-DF (three seeds), 3rdBench (two), and GenImage (one). The decision layer is the stronger of the two where the expert evidence is weakest, at a cost of 11.7 ms per image, and one forward pass also returns its per-hypothesis, auditable outputs.
In the revised PDF, the fusion heads are in Table 3 and Appendix I, the experts and the gate floor are in Appendix E (Tables 10 and 11), and the per-seed and per-confidence analysis is in Appendix F (Tables 12 to 16).
We are glad to clarify any of this.
Thanks for the reviewer's precise and constructive suggestions (Part 1/2)
We thank the reviewer for a precise and constructive review. The review does more than ask whether the VLM is needed. It specifies the experiment that answers the question: the same experts, the same training data, controlled access to evidence features and auxiliary supervision, and both AUROC and cost. We ran this experiment during the discussion period and report it below.
W1 / Q1: We agree that this comparison was missing. We have now run it as a new experiment.
Setup. All baselines train on the same 60,000 faces as PDM-Face. They read the identical 1,315-dimensional evidence vector that feeds PDM-Face's evidence tokens: the CNN features, the CLIP features, the radial spectrum, and the expert scalars. They use the same validation-based early stopping and temperature fitting. Each arm has three seeds, and the model is selected on validation data only. The four arms are:
Results. The multi-task MLP was the best arm on validation. The table gives the AUROC of PDM-Face minus this arm on identical images, with paired 95% confidence intervals (seed 0 of each).
Against the CLIP router, PDM-Face is significantly better on five of six benchmarks, for example GenImage (+0.090), FF++ (+0.063), DFDC (+0.037), and WildDeepfake (+0.012). The two are tied on 3rdBench.
Cost. All arms and PDM-Face share the same expert feature extraction, so the difference lies in the decision stage alone. The light fusion heads take under 0.1 ms per image. The VLM decision layer takes 11.7 ms per image (batched, with prefix-state branching, on one H100).
Reading. With the experts, the training data, the evidence features, and the auxiliary supervision all held fixed, the VLM decision layer still adds a significant gain on DFDC, FF++, and WildDeepfake. These are the three benchmarks where the expert layer is weakest (0.622, 0.622, and 0.749 AUROC). The evidence is ambiguous there, and the VLM reads the image jointly with it. On GenImage and 3rdBench, where the experts are already strong, a light head recovers the same accuracy. The added value of the VLM is therefore concentrated on the hard cases, at a cost of 11.7 ms per image. The decision layer also returns the eight hypothesis answers for each image as an auditable readout.
We will add this experiment to the revision (appendix), with all four arms, all seeds, and the cost table, and we will summarize it in the main text.