acceptodds
Withdrawn from ICLR 2027

Faithful Is Not Forensic: Robust Intervention-based Forensic Testing (RIFT) for Auditing Deepfake Detectors

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A deepfake detector can be accurate and faithfully explained while relying on evidence that does not account for the manipulation that changed an image's authenticity. We introduce RIFT, Robust Intervention-based Forensic Testing, a post hoc audit for frozen detectors that uses only scalar scores. Given a matched pristine and manipulated pair, RIFT applies the same intervention to a candidate region in both images. Manipulation Reliance (M) measures how much that region accounts for the matched authenticity contrast; Nuisance Instability (Q) is a complementary fixed-family stability diagnostic; and FSS is a conservative joint summary. Across six admissible detectors, the matched effect exceeds score-matched same-authenticity placebos by 0.198 to 0.666, with every paired interval above zero. The audit also withholds interpretation where raw metrics would mislead: a detector that passes the in-domain competence gate and shows a positive raw GT-over-control margin fails the placebo test (−0.035, 95% CI [−0.050, −0.019]), so no forensic claim is made for it. A shared-cue sanity check and geometry-matched spatial controls confirm that the matched construction behaves as defined, while operator changes, reserved nuisances, a failed prespecified stress test, and controlled FF++ residual transfer bound the admissible interpretation. RIFT requires no retraining, gradients, internal features, or architecture-specific hooks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
deepfake detection, forensic auditing, model auditing, explanation faithfulness, shortcut learning, counterfactual intervention, robustness, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mgPeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RIFT, a region-level audit for frozen deepfake detectors on matched real/fake pairs. RIFT measures how the positive detection gap changes when a supplied region is neutralized (M), whether this dependence remains stable under predefined perturbations (Q), and combines the two in FSS. Experiments on FF++ include shared-cue controls, spatial controls, same-authenticity placebos, and intervention operators. It audits supplied regions; it does not predict manipulation masks.

Critical strengths
  1. Matched real/fake pairs let the authors separate a region's effect on an individual prediction from its effect on the authenticity gap. This is useful for checking whether an explanation follows shared watermarks, backgrounds, or other cues present in both classes.
  2. The validation includes multiple spatial controls, placebos, and intervention operators. I found it useful that the authors report failed competence gates and limited Q/FSS gains; those results make the scope of the method easier to judge.
Critical weaknesses
  1. High M can still arise from an interaction between unchanged context and manipulated features, so the claim of forensic specificity needs qualification. The Faith comparison changes both the reference information and the measurement design, which makes their contributions difficult to separate.
  2. Several validation details are still hard to verify: score-matched placebo construction, the construction of GT and spatial controls, and the competence gates for the reserved nuisances. I also could not settle how Q should be read at low signal, and the aggregation behind Table 3 needs clarification.
Questions for authors
  1. Can interactions between unchanged shared context and manipulated features still produce a high M? If the existing checks do not exclude that case, the "forensic specificity" claim needs qualification. It would also help to state explicitly that M measures change relative to s_f, not the fraction of the original gap removed.
  2. The Faith comparison changes both the availability of a real-image reference and the measurement design. Could the authors add an explanation baseline that also uses matched real/fake pairs but is not equivalent to M, while holding samples, regions, interventions, and aggregation fixed? That would separate the value of the reference from the value of RIFT's design.
  3. Table 9 reports gate status for the primary nuisances, but the corresponding AUCs and gate status for the reserved nuisances are unclear. Please report these and, beyond the M-adjusted analyses in Table 7, check whether the Q associations persist across M ranges and when instability is measured in absolute terms.
  4. How were the score-matching variables chosen and what tolerance was used? Please clarify the pairing direction and the treatment of unmatched samples. Is Matched F-R in Table 6 computed with Eq. (4), using the same samples and weights as Table 1? I could not explain why TSRL is 0.1842 in one table and 0.083 in the other.
  5. What is the source of "exact manipulation support", and how is it binarized and aligned to the image and intervention grid? How many samples remain in each manipulation family after the strict translated controls are constructed? These details are needed to judge whether support definitions or sample selection could drive the spatial comparison.
  6. In Table 3, the strength-1.00 values give 0.447 − 0.024 = 0.423, whereas ΔFSS is reported as 0.413; other rows also differ. With common samples and weights, averaging paired differences should give the same point estimate. Please clarify any differences in samples or weights and reconcile the point estimates and intervals with Section 5.1.
Reviewer_EiWVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RIFT, a post-hoc auditing framework that distinguishes explanation faithfulness from forensic specificity in deepfake detection. Using matched pristine–manipulated image pairs, it proposes Manipulation Reliance (M) to measure regional contributions to prediction differences, Nuisance Instability (Q) to assess perturbation stability, and FSS to combine both. Experiments across multiple frozen detectors include controlled shortcut tests, GT-region comparisons, placebo checks, perturbation robustness, and controlled cross-domain transfer. The results primarily demonstrate the effectiveness of M in distinguishing known manipulation regions from spatial controls, while the additional benefits of Q and FSS remain limited.

Critical strengths
  1. The paper distinguishes explanation faithfulness from forensic specificity, offering a relevant perspective on evaluating deepfake detectors beyond prediction accuracy.
  2. RIFT combines matched-pair interventions with stability diagnostics and validity checks. The evaluation includes placebo tests, spatial controls, intervention robustness, and failed cases, providing systematic evidence for the proposed measurements under controlled condit
Critical weaknesses
  1. Faithfulness does not imply forensic validity, while M measures matched prediction differences rather than genuine forensic evidence. Its ability to distinguish forensic artifacts from correlated shortcuts, and thus its significance, remains unclear.
  2. M is essentially a paired intervention difference. The lack of comparisons with existing forensic explanation methods under a unified benchmark, alongside unequal information access between Faith and RIFT, weakens its novelty claims.
  3. The reliance on matched pairs and known GT masks favors GT-over-control separation. The experiments do not establish whether RIFT can distinguish genuine forensic artifacts from manipulation-correlated shortcuts within GT regions.
  4. Although RIFT aims to assess forensic specificity beyond detection accuracy, Table 1 shows identical detector rankings under ID AUC, M, and FSS. Given the limited cross-detector comparability of audit scores, controlled evidence that RIFT provides meaningful insights beyond detection competence remains lacking.
Questions for authors
  1. Can RIFT reliably distinguish genuine forensic evidence from manipulation-correlated shortcuts located within the GT regions, rather than primarily separating known manipulation masks from nonoverlapping controls?
  2. Given that M alone performs comparably to FSS, can the authors demonstrate a concrete case where Q provides additional forensic discrimination or corrects a failure of M?
  3. What is the methodological novelty of M beyond a straightforward difference of paired intervention effects? Can the authors provide information-matched comparisons with simple paired-perturbation baselines and existing forensic explanation methods?
  4. Can the authors demonstrate that RIFT captures meaningful forensic differences between detectors with comparable benchmark AUC, using controlled or within-detector evaluations that avoid comparing incompatible absolute audit scores?
Reviewer_L2hoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RIFT, a post-hoc auditing framework for frozen deepfake detectors based on interventions on matched pairs. Its core innovation lies in distinguishing between "evidence that influences the detector" and "evidence that explains changes in the authenticity of a match."

Critical strengths

The paper draws a clear distinction between "single-image influence" (of the "Faith" category) and "matched-pair authenticity contrast." This addresses a critical issue in deepfake forensics auditing: a specific region might exert a strong influence on a detector without actually accounting for the discrepancy between a "forged" image and its "authentic" counterpart.

Critical weaknesses
  1. The core task of the paper is to "audit whether a candidate region SS explains the change in detection confidence." However, in the experiments, SS is primarily derived from the ground-truth (GT) manipulation masks in the FF++ dataset, along with their corresponding geometric transformations. The "Faith" baseline is also computed using these same GT regions. This functions more as a "sanity check" given a known ground truth, rather than an audit of the outputs produced by actual explanation methods.
  2. Table 9 shows that among the six "full-control" detectors, all but DiffusionFake fail the "competence gate" for at least one primary nuisance condition; for TSRL, all four conditions fail.
  3. While the paper mentions methods such as Grad-CAM, Integrated Gradients, RISE, and Extremal Perturbation in the "Related Work" section, the experiments only compare against the perturbation-based "Faith" baseline. Although "Faith" serves as a reasonable baseline for single-image influence, it is insufficient to represent the entire spectrum of "faithfulness" evaluation methods. Given the strong title—"Faithful Is Not Forensic"—it would be appropriate to include at least one attribution-based method in the comparison.
  4. Table 15 indicates that only the WildDeepfake dataset features multiple detectors that achieve a "controlled Pair AUC" of ≥0.80\ge 0.80; for other domains, the Pair AUC is generally below 0.80. The authors transparently state that they only evaluate "controlled transfer" on the WildDeepfake dataset. While this is a reasonable scope, it also highlights the limited applicability of this cross-domain auditing approach. It should be clarified that the "controlled transfer" in RIFT does not represent a prediction of natural out-of-distribution (OOD) performance, but rather a controlled "stress test" based on constructing "FF++ residual contrasts" within the target domain.
Questions for authors
  1. Add an audit experiment using various explanation methods to generate candidate regions: employ techniques such as Grad-CAM, Integrated Gradients (IG), or RISE to produce sets SS, and compare how Faith and RIFT rank ground-truth (GT) regions, background areas, and shortcut regions.
  2. Clarify the added value of FSS over M: Table 11 shows that the two perform nearly identically; therefore, it is necessary to explain under what circumstances FSS is essential, or otherwise reclassify it as an optional summary metric.
  3. Include at least one additional attribution baseline, or explicitly specify the scope as "perturbation-based faithfulness."

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.