Hierarchical Adaptation and Causal Deconfounding for Generalizable Deepfake Detection

Existing deepfake detectors often suffer from poor generalization to unseen domains and manipulation types, which can be attributed to two intrinsic limitations. First, prevailing adaptation methods apply uniform fine-tuning across all backbone layers, overlooking the hierarchical representational heterogeneity of vision transformers: layers encoding fundamentally different information require distinct adaptation capacities, and a one-size-fits-all strategy can limit cross-distribution transfer. Second, models tend to rely on forgery-irrelevant confounding factors, inducing spurious correlations that obscure reliable forgery-related evidence; such shortcut dependencies can break down when the detector is transferred across datasets or manipulation pipelines. To address these two limitations, we propose a novel detection framework with two complementary modules: Adaptive Subspace Residual Tuning (ASRT) and Confounder-Suppressed Prototype Decomposition (CSPD). Specifically, to resolve the mismatch between uniform adaptation and hierarchical layer characteristics, ASRT exploits the singular-value spectrum of each layer to decompose pretrained parameters into a frozen dominant subspace and a trainable residual subspace. Unlike methods that impose a uniform decomposition rank across layers, it allocates layer-specific adaptation capacity according to each layer’s intrinsic representational property, better balancing prior preservation and forgery cue capture. To mitigate reliance on forgery-irrelevant confounders, CSPD constructs a learnable prototype pool to explicitly model recurring forgery-irrelevant nuisance factors and adaptively combines relevant prototypes to estimate and suppress their contributions in the feature space. This causally motivated feature-space residualization reduces spurious dependencies and encourages the detector to focus on forgery-related evidence. Extensive cross-dataset and cross-method experiments demonstrate that our method achieves the highest average AUC across the evaluated methods, showing strong generalization to unseen domains and manipulation types.