acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Adaptation and Causal Deconfounding for Generalizable Deepfake Detection

Abstract

Existing deepfake detectors often suffer from poor generalization to unseen domains and manipulation types, which can be attributed to two intrinsic limitations. First, prevailing adaptation methods apply uniform fine-tuning across all backbone layers, overlooking the hierarchical representational heterogeneity of vision transformers: layers encoding fundamentally different information require distinct adaptation capacities, and a one-size-fits-all strategy can limit cross-distribution transfer. Second, models tend to rely on forgery-irrelevant confounding factors, inducing spurious correlations that obscure reliable forgery-related evidence; such shortcut dependencies can break down when the detector is transferred across datasets or manipulation pipelines. To address these two limitations, we propose a novel detection framework with two complementary modules: Adaptive Subspace Residual Tuning (ASRT) and Confounder-Suppressed Prototype Decomposition (CSPD). Specifically, to resolve the mismatch between uniform adaptation and hierarchical layer characteristics, ASRT exploits the singular-value spectrum of each layer to decompose pretrained parameters into a frozen dominant subspace and a trainable residual subspace. Unlike methods that impose a uniform decomposition rank across layers, it allocates layer-specific adaptation capacity according to each layer’s intrinsic representational property, better balancing prior preservation and forgery cue capture. To mitigate reliance on forgery-irrelevant confounders, CSPD constructs a learnable prototype pool to explicitly model recurring forgery-irrelevant nuisance factors and adaptively combines relevant prototypes to estimate and suppress their contributions in the feature space. This causally motivated feature-space residualization reduces spurious dependencies and encourages the detector to focus on forgery-related evidence. Extensive cross-dataset and cross-method experiments demonstrate that our method achieves the highest average AUC across the evaluated methods, showing strong generalization to unseen domains and manipulation types.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Deepfake Detection,Domain Generalization,Hierarchical Adaptation,Causal Deconfounding,Feature Disentanglement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_H6rpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a generalizable deepfake detection framework consisting of Adaptive Subspace Residual Tuning (ASRT) and Confounder-Suppressed Prototype Decomposition (CSPD). ASRT uses layer-wise singular-value distributions to determine the trainable residual subspace of a pretrained CLIP ViT, while CSPD employs a learnable prototype pool to estimate and suppress forgery-irrelevant components in the feature space. Experiments on cross-dataset and cross-manipulation benchmarks, together with several ablation studies, are presented. The proposed framework is conceptually intuitive. However, I have concerns about the completeness and fairness of the baseline comparisons, the evidence supporting the causal deconfounding claim, and several methodological details. The experimental results do not sufficiently establish the claimed state-of-the-art generalization performance or the effectiveness of the proposed causal interpretation.

Critical strengths
  1. The paper is generally well-written and clearly organized, making the proposed method and experimental setup relatively easy to understand.
  2. The paper addresses the important problem of generalizable deepfake detection. Combining hierarchical parameter adaptation with confounder suppression is a reasonable and promising research direction.
Critical weaknesses
  1. Table 2 reports results for FA, but FA is missing from Table 1. Similarly, DFD-HR is included in Appendix Table 7 but omitted from the main cross-dataset comparison. Results for these methods are available in their original papers, and their weights are available for reproduction. These omissions raise concerns about the claimed superior cross-dataset performance. The authors should clarify why these comparisons are missing and provide a more comprehensive evaluation against recent methods.
  2. Table 2 evaluates BleFace, whereas Table 8 evaluates FSGAN. Moreover, most baseline methods are evaluated on eight subsets of DF40, while the proposed method considers only six. These inconsistencies make it difficult to assess the robustness of the reported results and raise concerns about potential selective reporting. The authors should justify the selection of manipulation methods and provide more consistent evaluations.
  3. CSPD aims to suppress forgery-irrelevant confounders, but the paper provides insufficient evidence that the learned prototypes actually capture nuisance factors such as identity and background. Moreover, the prototype usage patterns reported in the appendix suggest that they may also capture forgery-related information. Further analysis is needed to validate the claimed deconfounding effect.
  4. Equation (15) defines the weight loss as the negative mean of the residual singular values. Minimizing this term directly encourages the singular values to increase. Without an explicit constraint or a sufficiently justified balancing mechanism, this objective may encourage uncontrolled growth of the residual transformations rather than regularize them. The authors should clarify which quantities are optimized, how the singular values are constrained, and how this objective affects training stability.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer H6rp

We sincerely thank you for the time and effort you have devoted to reviewing our manuscript and for your thoughtful comments and insightful questions. Your feedback has helped us improve the clarity and completeness of the manuscript. We address your comments in detail below.

RH6rp-Q1: Coverage of recent methods in the cross-dataset comparisons

Thank you for raising this concern. We first selected methods closely related to our work and then included additional methods from the broader deepfake detection field. Both the frame-level and video-level comparisons included three methods published in 2025 and two published in 2026.Our intention was to cover a broader range of methods across the two tables to assess generalization, which led to differences in the baseline selections.

DFD-HR is a valuable reference for our work. Its cross-dataset evaluations also use different sets of comparison methods for frame-level and video-level results. In response to your concern, we have expanded the comparisons with recent methods in Tables 1 and 7 to provide a more comprehensive assessment of our method's generalization performance. For example, we have added DF-CBM(ECCV'26) to Table 1 and 7.

Evaluation levelMethodCDF-v2DFDCPDFDCDFDAvg.
Frame-levelDF-CBM (ECCV 2026)0.8300.7740.7320.8740.8025
Frame-levelOurs0.9030.8810.8510.9480.8958
Video-levelDF-CBM (ECCV 2026)0.8910.8160.7580.9270.8480
Video-levelOurs0.9610.9150.8730.9840.9333

RH6rp-Q2: Consistency of the DF40 evaluation subsets

We thank you for pointing out the inconsistency in the evaluation subsets. Our initial evaluation followed the setting reported in “When Detectors Forget Forensics: Blocking Semantic Shortcuts for Generalizable AI-Generated Image Detection,” which considers six manipulation subsets. When incorporating results from additional related work, we found that MAP-Mamba, published in TIFS in 2026, did not report results on FSGAN. We therefore replaced FSGAN with BleFace in Table 2, which accounts for the difference in subset selection between Tables 2 and 8.

In response to your comment, we have expanded the video-level evaluation in Table 2 to eight subsets. For the frame-level comparison in Table 8, some baseline papers report only video-level results, which cannot be directly used for a frame-level comparison. This limits our ability to extend the comparison to all eight subsets. In future work, we plan to reproduce the frame-level results of these baselines to enable a more comprehensive comparison.

RH6rp-Q3: Evidence that the prototypes capture nuisance factors

Thank you for raising this question. We address the same concern in our response to Reviewer gvj5, Q2, and our response to Reviewer W3j9, Q4. These responses discuss the interpretation of class-dependent prototype routing and the additional visualizations used to examine the effect of CSPD's subtraction operation. We refer the reviewer to those responses for details.

RH6rp-Q4: The role of Equation 15 and control of residual parameter growth

We thank you for highlighting this issue. The motivation for Equation 15 is that the residual components are initialized from directions associated with smaller singular values of the pretrained weights. We aim to strengthen the contribution of these directions so that they can participate effectively in adaptation for forgery detection. Equation 15 therefore serves to enhance residual adaptation capacity.

During optimization, the dominant component remains frozen, while the residual component is trainable. To counteract excessive parameter growth, we apply the weight decay used with the Adam optimizer, as specified in the manuscript, to the residual parameters, including the trainable singular values. Equation 15 encourages stronger residual transformations, while weight decay penalizes large parameter magnitudes, providing a counteracting mechanism.

The orthogonality loss serves a different purpose: it reduces redundancy among residual directions and does not directly constrain the magnitudes of the singular values.

We acknowledge that the original manuscript did not sufficiently distinguish the roles of these mechanisms. We have added a description of Equation (15) in lines 335–338 of the revised manuscript to clarify its role in enhancing residual adaptation capacity and explain how weight decay counteracts excessive parameter growth.

We thank you again for the helpful comments and suggestions.

Thank you for clarifying the original DF40 subset selection and expanding the video-level evaluation. The added DF-CBM baseline scores below Effort at the frame level, and the requested comparisons with FA and DFD-HR in Table 1 remain unresolved. DFD-HR already reports video-level AUC for all eight DF40 subsets and could be included in the expanded Table 2. For Table 8, I would like to see the proposed method compared on the same subsets with baselines whose frame-level results are available or can be reproduced. Given these gaps, I believe the claims of superior generalization should be stated more cautiously. I maintain my original rating.

Reviewer_W3j9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets generalization of face-forgery detectors to unseen datasets and unseen manipulation types, attributing the gap to two causes: (i) adaptation methods apply a uniform fine-tuning budget to all backbone layers despite the representational heterogeneity of ViT layers, and (ii) detectors latch onto forgery-irrelevant confounders (background, identity, style) that produce spurious correlations which break under distribution shift. Two modules address these. ASRT takes each 2D attention weight , picks a rank r as the smallest k whose cumulative singular-value energy exceeds a threshold, freezes the dominant part, and trains only the residual part. CSPD models nuisance factors with a pool of learnable prototypes; a softmax router selects the top-k, the renormalized weights form a sample-specific nuisance vector, and the classifier sees. Experiments use CLIP ViT-L/14, FF++ (c23) for training on a single RTX 3090, and evaluate cross-dataset (CDF-v2, DFDCP, DFDC, DFD) and cross-method (six DF40 methods: UniFace, e4s, FaceDan, FSGAN, InSwap, SimSwap), with frame- and video-level AUC. The method reports the best average in both protocols.

Critical strengths

1). The per-layer spectral rank is a cheap, sensible refinement of an established idea, and the paper ablates its parameterization honestly. 2). The ablation suite is complete for this subfield. 3). Two evaluation protocols, and some baselines are re-run. 4). The paper is candid about the limits of its own causal claim.

Critical weaknesses

1). The gain decomposes into mostly "adapt CLIP at all," and the two modules' marginal contributions are not shown to exceed noise. 2). "Hierarchical" is a post-hoc description, not a design, and the missing control is the obvious one. 3). The causal claim is not earned — and one of the paper's own figures argues against it. 4). Two tables disagree by 12.5 AUC points for what appears to be the same configuration, and Eq. 16 is malformed. Table 3 row 1 (no ASRT, no CSPD, i.e. CLIP + cross-entropy) reports CDF-v2 0.749, DFDCP 0.702, DFDC 0.704. Table 4 row 1 (L_cls only, no L_debias/L_weight/L_ot/L_fg) reports CDF-v2 0.874, DFDCP 0.837, DFDC 0.803. These should be the same model; they differ by 12.5, 13.5 and 9.9 points. 5). Scope mismatch between the motivation and the evaluation, on a single backbone, a single training set, and a single run. The abstract and introduction motivate the problem with diffusion and generative-AI realism, but every experiment uses face-swap or face-reenactment data (FF++ for training; six DF40 methods for cross-method testing) — no diffusion-generated faces, no fully synthetic imagery, no AIGC detector benchmark.

Questions for authors
  1. What are the trainable parameter counts for ASRT, Effort, and each adaptation baseline, and what happens to the comparison under a uniform-rank control matched to ASRT's average r and total trainable parameters? Without that control, the +2.3 frame / +0.9 video advantage over Effort is attributable to capacity rather than to layer-wise allocation. Please also report mean ± std over at least three seeds for Tables 1–3.
  2. What is T for the reported model, and what is r for each layer? Given the parameterization T = 1 − a(0.1)^n and the selected a=2, n=6, T ≈ 0.999998, which seems to leave a very small residual — please state r for a representative shallow/middle/deep layer, the resulting residual dimension, and whether W_res's U_res/V_res are frozen at their SVD initialization or trainable (L_ot implies the latter, which would make the "decomposition" an initialization rather than a fixed partition).
  3. How do you reconcile Table 3 row 1 (CDF-v2 0.749) with Table 4 row 1 (CDF-v2 0.874)?
  4. Fig. 6 shows prototype routing is strongly class-predictive. If prototypes model label-independent nuisance factors, why does their selection predict the label? Please add a probe experiment: train an identity classifier and a background/domain classifier on f and on f_d, and report whether f_d loses decodable identity/background information while retaining forgery discriminability.
  5. Why only face-swap/reenactment methods from DF40, given the stated threat model is generative AI? Please report results on diffusion-based or fully synthetic generators (DF40's diffusion subset, or an AIGC benchmark), and consider whether ASRT's spectral allocation behaves the same on a backbone whose pretraining objective is not contrastive (DINOv2, SigLIP).
1 comment
AuthorsOpenReview ↗

Response to Reviewer W3j9

We sincerely thank you for the time and effort you have devoted to reviewing our manuscript and for your thoughtful comments and insightful questions. Your feedback has helped us improve the clarity and completeness of the manuscript. We address your comments in detail below.

RW3j9-Q1: Trainable parameter counts, capacity-matched uniform-rank control, and multiple seeds

Thank you for raising these questions. We have added the trainable parameter counts and the capacity-matched uniform-rank comparison to Table 12. Further details are provided in the revised manuscript and our response to Reviewer gvj5, Q1.

Regarding experiments with multiple random seeds, most baseline results in our generalization evaluations are taken from the corresponding papers under the same evaluation protocol. These results generally do not include means and standard deviations across multiple runs. For all our experiments, we used the random seed 1024, following the seed used in most of the DeepfakeBench configurations we adopted, to maintain consistent experimental settings.

RW3j9-Q2: Layer-wise ranks and the residual subspace during training

The residual ranks and trainable parameter allocation for the shallow, middle, and deep layers are discussed in our response to Reviewer gvj5, Q1.

Regarding whether the residual components involved in LotL_{\mathrm{ot}} can change during training, we acknowledge that our original description was insufficiently precise. Before training begins, ASRT determines the dominant rank rr and the residual dimension from the singular-value spectrum of the pretrained weights in each layer. Once determined, these ranks and dimensions remain fixed throughout training; they are not progressively reallocated. We have added a clarification in lines 332–335 of the revised manuscript to explicitly state that the dominant rank and residual dimension of each layer are determined before training and remain fixed throughout optimization.

RW3j9-Q3: Differences between the first rows of Tables 3 and 4

We thank you for identifying this ambiguity. The two results correspond to different experimental settings, and the original manuscript did not explain this distinction sufficiently clearly.

Table 3 evaluates the contributions of ASRT and CSPD. Its first row is a CLIP baseline trained using only the classification loss, without either module. Subsequent rows introduce the corresponding modules together with their associated losses.

Table 4 evaluates the contributions of individual loss terms. Therefore, both ASRT and CSPD are retained in every row. The first row uses only the classification loss, while subsequent rows progressively introduce the auxiliary losses.

Thus, the first rows of the two tables use different model architectures, which explains the different results.We have added a clarification in lines 426–427 of the revised manuscript to explicitly distinguish these two experimental settings.

RW3j9-Q4: Class-predictive prototype routing and evidence of nuisance suppression

Regarding why prototype selection predicts the class label, the router operates on features represented in a high-dimensional space, where authentic and forged samples have different distributions. Authentic images share common characteristics and therefore tend to have similar representations. Forged images differ from authentic images and also vary across manipulation types, resulting in more heterogeneous representations.

Consequently, the nuisance components to be removed may differ between authentic and forged samples. Prototype selection can therefore carry class information. We interpret this behavior as consistent with the prototype pool adaptively selecting different prototypes to suppress sample-dependent nuisance components.

To further examine the effect of the subtraction operation, we have added feature visualizations to the revised manuscript. Please see our response to Reviewer gvj5, Q2, for details.

RW3j9-Q5: Evaluation scope and allocation behavior on other backbones

Thank you for this suggestion. Many deepfake detection methods, such as Effort and Tutor, evaluate face-swap and face-reenactment methods from DF40. Our evaluation follows this setting. We plan to extend our method to jointly address manipulated-face detection and detection on AIGC benchmarks in future work.

For DINOv2 and SigLIP, ASRT exhibits similar allocation behavior. ASRT allocates ranks according to the spectral characteristics of the backbone itself. We hypothesize that this similarity may be related to the shared ViT architecture of the visual backbones considered.

We thank you again for the constructive comments and suggestions.

Reviewer_gvj5OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a CLIP-based deepfake detector combining Adaptive Subspace Residual Tuning (ASRT) and Confounder-Suppressed Prototype Decomposition (CSPD). ASRT partitions attention weights into frozen dominant and trainable residual components using matrix-specific singular-value spectra. CSPD subtracts an input-dependent mixture of learned prototypes from image features. The method reports strong cross-dataset and cross-method results, although the proposed mechanisms need more direct validation.

Critical strengths

The empirical results are consistently encouraging. The method improves average cross-dataset frame AUC from Effort’s 0.8725 to 0.8958 in Table 1 and achieves 0.9600 average video AUC in the cross-method evaluation. Additional appendix results extend the evaluation beyond the main comparisons.

The framework addresses two relevant challenges through relatively simple components. The singular-spectrum rule provides a concrete approach to allocating adaptation capacity, while the prototype module models sample-dependent components rather than subtracting a single shared feature vector.

The ablations provide useful evidence of empirical contributions from both modules. Table 3 evaluates their individual and combined effects, and Table 6 compares CSPD with alternative suppression strategies using the same ASRT backbone. Further studies examine prototype count, routing sparsity, and individual loss terms.

Critical weaknesses

The evidence for ASRT’s hierarchical allocation does not measure the quantity used by the method. Section 3.1 points to Figure 5 as evidence of layer-wise residual ranks, but the figure plots effective ranks of trained Q/K/V activations. Equation 2 instead allocates capacity using pretrained weight spectra. Reporting the actual allocated ranks and trainable parameter counts, alongside a uniform or shuffled allocation matched for total capacity, would establish whether hierarchy-aware allocation explains the improvement.

CSPD’s interpretation as nuisance suppression is insufficiently validated. Equation 10 attracts authentic features toward prototypes while limiting forged-feature similarity, which can produce class-conditioned representations without separating identity, background, or style. Figure 6 demonstrates class-dependent routing but does not establish what information is removed. The paper appropriately acknowledges that CSPD is not an exact causal intervention; nevertheless, controlled nuisance changes or probes before and after subtraction would directly test its proposed mechanism.

Model-selection and implementation details leave uncertainty about the generalization estimates. The appendix identifies preferred prototype counts, Top-k values, and subspace thresholds through cross-dataset performance, but does not clearly explain whether these decisions use separate validation data or are post hoc sensitivity analyses. Checkpoint selection, numerical values for the margin mm and diversity threshold τ\tau, and variability across training runs also need clarification.

Questions for authors

Please address the points raised in the weaknesses above, especially the actual rank allocation, evidence of nuisance suppression, and model-selection protocol.

1 comment
AuthorsOpenReview ↗

Response to Reviewer gvj5

We sincerely thank you for the time and effort you have devoted to reviewing our manuscript and for your thoughtful comments and insightful questions. Your feedback has helped us improve the clarity and completeness of the manuscript. We address your comments in detail below.

Rgvj5-Q1: Actual rank allocation and capacity-matched comparison

Thank you for pointing out the distinction between the effective ranks of trained Q/K/V activations and the pretrained weight spectra used for allocation. To directly report the capacity allocated by ASRT, we have added Table 12(line 846-861) to the revised manuscript. The stage-wise allocation is summarized below:

StageLayersSum of residual dimensionsTrainable SVD parameters
Shallow1–8266540.6K
Middle9–1656113.8K
Deep17–2450101.6K

The actual allocated residual ranks for the shallow, middle, and deep layers: 266, 56, and 50, respectively.These allocations are consistent with our description that shallow layers receive more trainable parameters.

We have also added a uniform allocation control matched for the total trainable parameter budget. The Frame-AUC results are:

AllocationCDF-v2DFDCPDFDCAverage
Uniform0.8970.8690.8380.8680
Ours0.9030.8810.8510.8783

Under this capacity-matched comparison, our hierarchy-aware allocation improves average AUC by 1%. This result supports the contribution of hierarchy-aware allocation to the performance improvement, with total trainable capacity held constant.

Rgvj5-Q2: Evidence of nuisance suppression

To examine the effect of the subtraction operation, we have added visualizations comparing the model with and without CSPD in Figure 7(lines 946-969) of the revised manuscript. These visualizations show that the model focuses more on manipulated facial regions when CSPD is used.

Regarding how Equation 10 guides prototype learning, authentic and forged images contain both shared characteristics and distinct information. The margin mm allows forged features to retain some similarity to the prototypes, accommodating the authentic-image characteristics shared by both classes. Specifically, the forged-sample term imposes a penalty only when the maximum similarity between a forged feature and the prototypes exceeds mm. Together with the attraction of authentic features toward the prototypes, this constraint is intended to encourage the prototypes to capture forgery-irrelevant characteristics.

Rgvj5-Q3: Model-selection protocol and implementation details

We thank you for highlighting the need to clarify the model-selection protocol. In Section 4.1(lines 370-371) of the revised manuscript, we explicitly report the following settings: the number of prototypes is 20, the Top-k routing parameter is k=2k=2, the subspace threshold parameters are a=2a=2 and n=6n=6, the margin is m=0.8m=0.8, and the diversity threshold is τ=0.3\tau=0.3.These hyperparameter values were determined through post hoc analysis of the cross-dataset performance reported in the appendix.

For checkpoint selection, we train the model on the FF++ training set and select the checkpoint based on validation-set performance. We then use the selected checkpoint for cross-dataset and cross-method evaluations. Results on the target test sets are not used for checkpoint selection.

We thank you for their positive and constructive feedbacks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.