Most of the Ablation Gain Was the Baseline's Training Budget: A Placebo-Controlled Audit of Evidence-Guided Deepfake Detection
Abstract
Evidence-guided detectors of AI-generated images fuse a forensic evidence map into an image backbone, and the ablation row "without the evidence branch" is read as the contribution of the map. That row is the sum of three terms: how far the baseline was trained, what the fusion module adds without any map, and what the content of the map adds. We measure each term separately. On a frozen cross-generator test set of 22,999 images and five training seeds, our detector beats a linear probe capped at 100 epochs by 18.35 points of true-positive rate at 5% false-positive rate. Training the probe to its own stopping criterion removes 15.77 of those points; a placebo arm, the same module retrained on all-zero maps, accounts for 1.98 more; the content of the map adds 0.60 (two-level 95% interval [−0.96, +2.20]). Against the converged probe the fused detector shows no gain distinguishable from zero on clean images and is 0.033 AUC lower under blur; for the module retrained without a map, 0.029 AUC lower, a probe on the module's own input attributes half of that loss, and nearly all of the corresponding TPR loss, to the module's parameters. The map share is the one quantity that does not move: across four placebos, three training budgets and two resampling schemes the upper end of its macro-AUC interval is at most +0.0103, and every reported map-share interval that excludes zero does so on the positive side. Two of five seeds nevertheless read the map, and those are the two seeds that lose most under blur. For a published method, LaRE², the same ablation read under three seeds and two checkpoint rules spans −0.016 to +0.060 AUC and changes sign for two of three seeds. We define the ablation resolution of a protocol and show that a single run cannot separate these terms. Our evidence covers one detector we built and one published method; it shows that the problem exists, not how common it is.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.