acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Vision Foundation Models for High-Fidelity Forgery Detection by Diagnosing Optimization Collapse

Abstract

Vision Foundation Models (VFMs) offer a promising route to generalizable deepfake detection, but it remains unclear whether representations learned through general-purpose pretraining retain the subtle, transferable forensic cues needed to detect high-fidelity forgeries. We expose this representational weakness through Optimization Collapse, in which linear probes trained on frozen VFM features for high-fidelity forgery detection degrade to chance-level performance even under small adversarial parameter perturbations. We quantify this vulnerability using the perturbation radius at which collapse occurs, termed the Critical Optimization Radius (COR). Theoretical analysis, supported by empirical evidence, shows that the COR is positively associated with the Gradient Signal-to-Noise Ratio (GSNR), a measure linked to generalization, motivating the use of COR as a diagnostic for assessing the strength of transferable forensic cues retained in VFM representations. We validate that higher forgery fidelity exacerbates this optimization vulnerability in both cross-manipulation comparisons and controlled within-manipulation interpolation studies, as reflected in lower COR and poorer detector generalization. Layer-wise analysis further reveals the mechanism underlying Optimization Collapse: for high-fidelity forgeries, COR and GSNR progressively decline with depth, indicating that transferable forensic cues are increasingly attenuated as features propagate through the frozen backbone. Reducing the perturbation radius can avoid collapse but does not recover the attenuated forensic cues. To mitigate this collapse by improving GSNR, we propose the Contrastive Regional Injection Transformer (CoRIT), which uses feature differences as a forward-computable proxy for the gradient signal. CoRIT pools token features based on regional gradient-proxy alignment to reduce gradient-proxy variance, injects the pooled features into auxiliary tokens to preserve regional signals, and fuses intermediate and final-layer representations. CoRIT keeps the backbone frozen and optimizes only a lightweight classification head. Experiments show that CoRIT increases both COR and GSNR while achieving the highest average AUC among the compared methods in cross-manipulation and cross-dataset evaluations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Deepfake Detection, Vision Foundation Models, Optimization Theory, Generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HSAUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the sensitivity of linear probes on frozen vision foundation models to adversarial parameter perturbations during SAM training. It introduces the Critical Optimization Radius (COR), relates an analytical approximation to the gradient signal-to-noise ratio (GSNR), and interprets layer-wise changes as evidence of attenuated forensic cues. The proposed CoRIT uses original–SBI feature differences to refine regional token selection, injects pooled features into auxiliary tokens, and combines intermediate and final-layer representations. Experiments evaluate cross-manipulation and cross-dataset detection while training only a lightweight classification head.

Critical strengths
  1. The observed variation in SAM sensitivity across manipulation types, backbone layers, and model families is interesting. The radius sweeps and layer-wise analyses provide a concrete empirical phenomenon that merits investigation as a possible diagnostic of forensic representations.

  2. CoRIT achieves meaningful gains with a frozen backbone. Component ablations support the usefulness of regional injection, refinement, and intermediate-layer fusion, while head-only training offers practical memory savings.

Critical weaknesses
  1. The central interpretation of COR as a diagnostic of transferable forensic information is insufficiently established. Lemma 1 supplies a sufficient descent condition; exceeding its bound does not imply collapse or chance-level classification. Appendix B.4 acknowledges this distinction, but the subsequent representation-level conclusions require stronger validation of the analytical approximation against the observed collapse threshold. COR also depends on parameterization: rescaling features and inversely rescaling a linear head preserves predictions while changing the effect of a fixed parameter-space perturbation. The reported layer-wise trends therefore do not adequately separate loss of forensic information from scale and conditioning effects. The COR–GSNR theorem is a decomposition of the defined quantities, whose diagnostic interpretation additionally depends on the statistical-geometric factor remaining comparable across settings. These gaps directly affect the paper's principal mechanistic contribution, beyond the empirical usefulness of CoRIT.

  2. The connection between the SBI-based gradient proxy and the gradients relevant to the actual detector needs stronger validation. Appendix C's Taylor expansion and Cauchy–Schwarz argument identify favorable alignment, but do not establish that observed SBI residuals align with task gradients. Its Jacobian argument concerns hypothetical backbone gradients, whereas CoRIT optimizes only the classification head. Consequently, reduced regional proxy variance does not by itself establish improved parameter-gradient GSNR. The component ablations provide encouraging evidence that the design is useful, but direct measurements are needed to distinguish the proposed gradient-signal explanation from the benefits of regional aggregation and multi-layer feature access.

Questions for authors
  1. Assessment-critical validation: After controlling for feature scale and selecting optimizer settings on validation data, does COR retain predictive value for cross-domain AUC beyond ordinary linear-probe performance? How closely does the analytical approximation track the empirical collapse threshold?

  2. Mechanism validation: Can the authors measure alignment between the SBI residual and the relevant feature gradient, and connect regional pooling to changes in the classifier-head gradient GSNR actually used in training?

4 comments

We sincerely thank the reviewer for the constructive validation requests. We have added discussions and experiments that clearly delineate the conditions under which our theory holds. Below, we respond point by point. Each point starts with a short answer, followed by details. We hope these responses address your concerns.

W1 & Q1: Does exceeding the analytical bound actually match the observed collapse?

Answer: The analytical COR keeps the empirical ranking on all four manipulations, nearly equals it in the collapse regime (NT: 0.007 vs. 0.01, F2F: 0.069 vs. 0.07), and correlates with it at Pearson 0.94–0.99 across layers and backbones.

The analytical COR tracks the measured collapse threshold.

CORNTF2FFSDF
Analytical0.0070.0690.1670.189
Empirical0.010.070.891.40

The analytical COR preserves the empirical ordering of the four manipulations and is closest to the empirical value where collapse occurs (NT: 0.007 vs. 0.01, F2F: 0.069 vs. 0.07). It is below the empirical COR for every manipulation, which is what the theory predicts. Lemma 1 gives a sufficient condition for stable descent, ρ<∥∇Lt∥Lt,\rho<\frac{\lVert\nabla\mathcal{L}_t\rVert}{L_t}, so training is guaranteed to be stable below this bound and may remain stable above it.

NT and F2F collapse at the standard radius, and for them the two values nearly coincide. This matches Appendix B.3, where the bound is tight in the collapse regime (COR < 0.05). The gap appears only for FS and DF, whose COR lies far above the collapse regime.

Across layers and backbones, the analytical and empirical COR of the four manipulations remain highly correlated:

SettingPearson
CLIP-L, layer 50.97
CLIP-L, layer 150.96
CLIP-L, last layer0.94
DINOv3-L, last layer0.99
SigLIP2-L, last layer0.98

W1 & Q1: Are COR differences and layer-wise trends only effects of feature scale and parameterization?

Answer: COR is a training-time diagnostic like GSNR: its absolute value depends on scale, but the ranking and relative values stay stable across all tested normalizations, learning rates, feature scales, and optimizers. Our conclusions therefore remain valid.

We acknowledge that COR is not reparameterization-invariant, but its relative values under a fixed setting are stable. Feature normalization barely changes the absolute COR:

NormNTF2FFSDF
Raw0.010.070.891.40
L20.010.121.261.35
RMS0.0080.061.031.14
Δ_norm/Δ_forgery0.1%4.3%27%19%

The last row divides the largest change of each manipulation across the three normalizations by the COR gap across manipulations under Raw. Normalization shifts each manipulation by at most 27% of this gap and by 13% on average. These shifts are small relative to the gaps between manipulations, and the ranking NT < F2F < FS < DF is preserved under all three normalizations.

Learning rate, feature scale, and optimizer change the absolute COR, but each change moves all four manipulations in the same direction (no COR decreases as the scale grows):

lrNTF2FFSDF
5e-40.0040.050.541.24
1e-30.010.070.891.40
2e-30.0140.121.491.67
Feature scaleNTF2FFSDF
×0.250.0020.060.770.85
×0.50.0080.060.830.96
×10.010.070.891.40
×20.010.131.321.62
×40.030.281.832.57
Optimizer (CLIP-ViT-L)NTF2FFSDF
SAM-SGD0.010.070.891.40
SAM-AdamW0.040.091.232.08

We therefore also report the relative COR Zd=CORd−COR‾sCOR,Z_d=\frac{\mathrm{COR}_d-\overline{\mathrm{COR}}}{s_{\mathrm{COR}}}, where COR‾\overline{\mathrm{COR}} and sCORs_{\mathrm{COR}} are the mean and standard deviation of COR over the four manipulations under the same setting. The relative COR stays stable:

lrNTF2FFSDF
5e-4-0.91-0.820.161.57
1e-3-1.00-0.900.511.39
2e-3-1.06-0.930.881.11
Δ_lr/Δ_forgery6.3%4.6%30%19%
Feature scaleNTF2FFSDF
×0.25-1.07-0.920.891.10
×0.5-1.05-0.930.841.14
×1-1.00-0.900.511.39
×2-1.07-0.900.781.20
×4-1.08-0.850.621.31
Δ_scale/Δ_forgery3.3%3.3%16%12%
Optimizer (CLIP-ViT-L)NTF2FFSDF
SAM-SGD-1.00-0.900.511.39
SAM-AdamW-0.96-0.910.441.44
Δ_opt/Δ_forgery1.7%0.4%2.9%2.1%

Every setting change moves the relative COR by at most 30% of the gap across manipulations and by 8% on average. Cross-setting effects are too small to reorder the manipulations, and the ranking NT < F2F < FS < DF holds in every setting. The conclusions of the paper are therefore unaffected.

Layer-wise trends are not a scale effect. All layer-wise COR values are measured by feeding each layer's features into the same head under the same setting. Rescaling moves all four manipulations in the same direction. Across layers, however, COR rises for DF and FS but falls for NT and F2F (Figure 1(b)). A common scale or conditioning change cannot produce opposite trends, so the attenuation reflects the loss of forensic information. Across layers, γ also varies far less than GSNR (see W2), so the layer-wise COR trend is driven mainly by GSNR.

Our Main Position

COR is a training-time diagnostic, like GSNR [1] (Liu et al., ICLR 2020), and it captures differences that static measures cannot. All manipulations share the same frozen backbone, so static measures of its weights, such as parameter norms, are identical for every forgery type and cannot separate them. COR and GSNR are measured while training on each dataset, so they reflect how well the features support learning on that data. The cost is that their absolute values depend on training hyperparameters. Under the same training setting, however, the ranking of the relative values across the four manipulations stays consistent, so the conclusions of the paper are unaffected.

Q1: With optimizer settings selected on validation data, does CoRIT still go beyond an ordinary linear probe?

Answer: Yes. When UniFD (a linear probe) and CoRIT each select their own optimizer setting and checkpoint on validation data, CoRIT still leads by 4.9 (cross-manipulation) and 10.5 (cross-dataset) AUC points.

We let UniFD and CoRIT each select its optimizer setting (plain SGD, plain AdamW, or SAM-SGD) and checkpoint on the source validation set. UniFD selects plain SGD, and CoRIT selects SAM-SGD with ρ = 0.07. Average AUC over the four training sources:

MethodCross-manipulationCross-dataset
UniFD81.978.4
CoRIT86.888.9

COR also predicts where CoRIT helps most: in Table 1, the gains are largest on the low-COR manipulations F2F (+12.5) and NT (+8.2).

W2: Does the COR–GSNR decomposition carry diagnostic meaning only if γ stays comparable?

Answer: Empirically, γ is stable across training steps (Appendix B.5.1, Figure 7), and across manipulations, layers, and backbones √γ varies less than √GSNR (at most 36% vs. 146%). Moreover, the variation of γ does not make the rankings of COR and GSNR disagree.

Eq. (4) is an identity, so its value lies in attribution: it splits COR into gradient-signal quality (GSNR) and a noise-to-curvature scale (γ): COR=γ⋅GSNR.\mathrm{COR}=\sqrt{\gamma}\cdot\sqrt{\mathrm{GSNR}}. Intuitively, in the physical picture behind momentum-based optimizers such as Adam [2] (Kingma & Ba, ICLR 2015), the gradient acts as a force that moves the parameters. Taking the mean gradient as a force F=∣∇L∣F=\\|\nabla\mathcal{L}\\|, gradient noise as σ=Tr⁡(Cov⁡[g])\sigma=\sqrt{\operatorname{Tr}(\operatorname{Cov}[g])}, and curvature as a stiffness k=λmax⁡(H)k=\lambda_{\max}(H):

  • COR =F/k=F/k is the displacement driven by the true signal.
  • γ=σ/k\sqrt{\gamma}=\sigma/k is the displacement driven by noise.
  • GSNR=F/σ\sqrt{\mathrm{GSNR}}=F/\sigma is their ratio.

SAM [3] (Foret et al., ICLR 2021) shows that solutions that withstand larger parameter perturbations tend to generalize better, so a larger COR indicates that the representation supports better generalization. γ\sqrt{\gamma} is the displacement scale of gradient noise, which mainly comes from random sampling differences within a batch. With a large batch, this variance is relatively stable, so intuitively γ does not change the rankings of COR and GSNR.

We measured how much each factor varies.

  • Across training steps: the numerator and denominator of γ change together, so γ stays far more stable than either term (Appendix B.5.1, Figure 7).
  • Across manipulations, layers, and backbones: since COR =γ⋅GSNR=\sqrt{\gamma}\cdot\sqrt{\mathrm{GSNR}}, we compare the variation of γ\sqrt{\gamma} with that of GSNR\sqrt{\mathrm{GSNR}}:
Comparison√GSNR√γCOR
Manipulations (NT, F2F, FS, DF)35%11%50%
Layers (5, 10, 15, 20, 24)62%22%98%
Backbones (CLIP, DINOv3, SigLIP2)146%36%234%
  • Ranking: the empirical COR, the analytical COR, and GSNR all follow NT < F2F < FS < DF.

In every comparison, √γ varies only about one quarter to one third as much as √GSNR, and GSNR accounts for 70–75% of the variation in COR. γ therefore affects COR but stays comparable across settings, and it does not alter the ranking in our measurements. The diagnostic reading of COR is carried mainly by GSNR.

W3 & Q2: Do SBI residuals align with task gradients, and does regional pooling improve the head-gradient GSNR used in training?

Answer: The SBI residual shows a fairly strong positive correlation with the feature gradient (Spearman 0.51–0.61 across layers, vs. ≈0 when the residual is shuffled or replaced by a random vector). The Jacobian argument in Appendix C concerns the classification head, and the full model has the highest head-gradient GSNR (0.065 vs. 0.046–0.050 without any one module).

W3-1 & Q2-1: SBI Residual vs. Feature Gradient

The SBI residual is aligned with the feature gradient at every layer, and random controls show no alignment. Spearman correlation between the activation strengths of the feature gradient and of (i) the SBI residual, (ii) the SBI residual randomly shuffled, and (iii) a random vector:

LayerSBI residualGlobal shuffleRandom vector
50.58-0.0004-0.0009
100.61+0.0006-0.0001
150.59-0.0010-0.0001
200.51-0.0003+0.0003

The moderate-to-strong correlation disappears once the spatial correspondence is broken, so it is not a chance effect.

W3-2 & Q2-2: Regional Pooling and Head-gradient GSNR

The Jacobian argument in Appendix C concerns the classification head, not hypothetical backbone gradients. When we measure COR/GSNR at layer ll, the layer-ll features are fed directly into the classification head without passing through later layers. Regional proxy variance thus approximates the variance of the head-parameter gradient.

We directly measured the GSNR of the classification-head gradient used in training (average over four manipulations):

RSIRRMHRIHead GSNR
✓✓0.046
✓✓0.047
✓✓0.050
✓✓✓0.065

Removing any module lowers the head GSNR, which directly links the regional design to the gradient signal actually used in training, beyond the benefit of regional aggregation or multi-layer access alone.

References

[1] Jinlong Liu, Yunzhi Bai, Guoqing Jiang, Ting Chen, and Huayan Wang. Understanding Why Neural Networks Generalize Well Through GSNR of Parameters. In ICLR, 2020.

[2] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015.

[3] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-Aware Minimization for Efficiently Improving Generalization. In ICLR, 2021.

Reviewer_dGU8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates SAM training collapse when probing frozen vision foundation models for face forgery detection. It introduces Critical Optimization Radius (COR), relates it to gradient signal-to-noise ratio, and proposes CoRIT, which combines regional feature refinement, token injection, and intermediate-layer fusion. Experiments report improved cross-manipulation and cross-dataset generalization, supported by component and backbone ablations.

Critical strengths

1.The reported improvements span several evaluation settings. CoRIT achieves 86.8% average cross-manipulation AUC and 88.9% average classical cross-dataset AUC, compared with 84.2% and 86.4% for the strongest respective comparison methods. Component ablations support the practical contribution.

2.Training only the classification head makes the approach attractive computationally. The reported training memory reduction relative to ForAda is approximately 90%.

Critical weaknesses

1.COR’s interpretation as a representation diagnostic is insufficiently established. Equation (2) provides a sufficient descent bound, while Equation (4) is an algebraic decomposition; neither alone establishes the threshold for chance-level performance or predicts unseen-dataset generalization. Feature-scale controls are needed before interpreting COR differences across layers and backbones as attenuation of forensic information.

2.Baseline optimization controls remain unclear. The paper demonstrates that reducing SAM’s radius can restore learning, but does not clearly document independently validation-tuned radii for every baseline. Sharing the SAM optimizer is insufficient to isolate CoRIT’s contribution from differences in optimizer sensitivity.

Questions for authors

1.How accurately does analytical COR predict the measured collapse threshold, and do its rankings remain stable after feature normalization or rescaling?

2.Do CoRIT’s gains persist against baselines with independently validation-tuned SAM radii, learning rates, and stopping criteria, and against ordinary SGD/AdamW?

3.Please clarify test-time SBI generation and explain the differing CLIP-L UniFD results in Tables 2 and 8. These details affect reproducibility and interpretation of the baseline improvements.

3 comments

We sincerely thank the reviewer for recognizing the consistent gains and the efficiency of head-only training, and for the constructive questions. Below, we respond point by point. Each point starts with a short answer, followed by details. We hope these responses address your concerns.

W2 & Q2: Do CoRIT's gains hold against independently tuned radii and ordinary SGD/AdamW?

Answer: CoRIT's gain does not come from a favorable radius. Even at ρ = 0, i.e., plain SGD without SAM, CoRIT reaches 86.7% average cross-dataset AUC, above every baseline in Table 2 (best: 86.4%). When each method selects its own optimizer setting on validation data, CoRIT still leads UniFD by 4.9 (cross-manipulation) and 10.5 (cross-dataset) points. We do not fix collapse by lowering ρ. Instead, we use the SAM radius as a probe to diagnose the generalization problem and as the motivation for our method design, and CoRIT improves the generalization of the representation (i.e., raises COR).

CoRIT's gain does not come from a favorable radius. CoRIT under different ρ (average cross-dataset AUC, Table 2 setting):

ρAvg. AUC (%)
0 (plain SGD)86.7
0.0587.5
0.07 (default)88.9
0.0888.4

At ρ = 0, CoRIT already outperforms all methods in Table 2 (Section 5.2), including UniFD (76.9), ForAda (83.8), and ForensicConcept (86.4). Hence the gain over baselines comes from the representation, not from optimizer sensitivity. SAM then adds a further 2.2 points, which is possible only because CoRIT raises COR above the radius in use. For a baseline with a tiny COR, a smaller ρ avoids collapse but gives up SAM's generalization gain. We attribute this small COR to the limited generalization of the representation itself.

Independently selected settings. We further let UniFD and CoRIT each select its optimizer setting (plain SGD, plain AdamW, or SAM-SGD) and checkpoint on the source validation set. UniFD selects plain SGD, and CoRIT selects SAM-SGD with ρ = 0.07. Average AUC over the four training sources:

MethodCross-manipulationCross-dataset
UniFD81.978.4
CoRIT86.888.9

CoRIT keeps a clear lead when each method uses its own best setting.

Our Main Position

COR is our diagnostic probe and the motivation for our method design, not our fix. Lowering ρ avoids collapse but gives up SAM's generalization gain, so the largest radius a representation can tolerate tells how much transferable forensic cue it carries. This new perspective turns an optimization failure into a representation-level diagnosis of frozen VFMs, and CoRIT is designed to raise COR rather than to lower ρ. We also delineate the conditions under which our theory holds and verify them experimentally, and the original conclusions of the paper remain unchanged.

W1: Why does COR indicate generalization, if Eq. (2) is only a sufficient bound and Eq. (4) a decomposition?

Answer: Our claim rests on three links: intuition, theory, and experiment.

  1. Intuition. ρ is the key SAM hyperparameter that makes the parameters robust to perturbations and improves generalization. COR, the critical ρ, therefore measures in reverse how much generalization the representation supports. A small COR means the features cannot support the flatness-seeking that improves generalization.
  2. Theory. Eq. (4) links COR to GSNR [1] (Liu et al., ICLR 2020), an established measure of generalization, by ρ² = γ·GSNR. Appendix B.5.1 shows that γ is much more stable than its numerator and denominator (Figure 7), and across manipulations, layers, and backbones γ varies far less than GSNR, so the ranking does not change. Figure 5 confirms that COR and GSNR follow the same layer-wise trend. The chance-level threshold itself is measured directly by the ρ-sweep over 10 forgery techniques (Figure 2(a)), and the analytical COR from Lemma 1 tracks it (see Q1).
  3. Experiment. For all four methods in Table 1, held-out AUC follows DF > FS > F2F > NT, the same order as COR. CoRIT gains most where COR is lowest (+12.5 on F2F, +8.2 on NT), and the fusion layer with the highest COR gives the best AUC (layer 18: COR 2.47, AUC 88.9).

W1 & Q1: Do COR rankings survive feature normalization and rescaling?

Answer: After L2/RMS normalization or rescaling features from ×0.25 to ×4, the COR ranking is unchanged in all settings.

Feature-scale and optimizer control. Feature normalization barely changes the absolute COR:

NormNTF2FFSDF
Raw0.010.070.891.40
L20.010.121.261.35
RMS0.0080.061.031.14
Δ_norm/Δ_forgery0.1%4.3%27%19%

The last row divides the largest change of each manipulation across the three normalizations by the COR gap across manipulations under Raw. Normalization shifts each manipulation by at most 27% of this gap and by 13% on average. These shifts are small relative to the gaps between manipulations, and the ranking NT < F2F < FS < DF is preserved under all three normalizations.

Learning rate, feature scale, and optimizer change the absolute COR, but each change moves all four manipulations in the same direction (no COR decreases as the scale grows):

lrNTF2FFSDF
5e-40.0040.050.541.24
1e-30.010.070.891.40
2e-30.0140.121.491.67
Feature scaleNTF2FFSDF
×0.250.0020.060.770.85
×0.50.0080.060.830.96
×10.010.070.891.40
×20.010.131.321.62
×40.030.281.832.57
Optimizer (CLIP-ViT-L)NTF2FFSDF
SAM-SGD0.010.070.891.40
SAM-AdamW0.040.091.232.08

We therefore also report the relative COR Zd=CORd−COR‾sCOR,Z_d=\frac{\mathrm{COR}_d-\overline{\mathrm{COR}}}{s_{\mathrm{COR}}}, where COR‾\overline{\mathrm{COR}} and sCORs_{\mathrm{COR}} are the mean and standard deviation of COR over the four manipulations under the same setting. The relative COR stays stable:

lrNTF2FFSDF
5e-4-0.91-0.820.161.57
1e-3-1.00-0.900.511.39
2e-3-1.06-0.930.881.11
Δ_lr/Δ_forgery6.3%4.6%30%19%
Feature scaleNTF2FFSDF
×0.25-1.07-0.920.891.10
×0.5-1.05-0.930.841.14
×1-1.00-0.900.511.39
×2-1.07-0.900.781.20
×4-1.08-0.850.621.31
Δ_scale/Δ_forgery3.3%3.3%16%12%
Optimizer (CLIP-ViT-L)NTF2FFSDF
SAM-SGD-1.00-0.900.511.39
SAM-AdamW-0.96-0.910.441.44
Δ_opt/Δ_forgery1.7%0.4%2.9%2.1%

Every setting change moves the relative COR by at most 30% of the gap across manipulations and by 8% on average. Cross-setting effects are too small to reorder the manipulations, and the ranking NT < F2F < FS < DF holds in every setting. The conclusions of the paper are therefore unaffected.

All layer-wise comparisons use the same head and training setting. Rescaling also moves all four manipulations in the same direction. In contrast, across layers the COR of DF and FS rises while that of NT and F2F falls (Figure 1(b)). These opposite trends cannot be produced by a common scale change, so the layer-wise attenuation reflects the loss of forensic information.

Q1: How accurately does the analytical COR predict the measured collapse threshold?

Answer: The analytical COR keeps the empirical ranking on all four manipulations, nearly equals it in the collapse regime (NT: 0.007 vs. 0.01, F2F: 0.069 vs. 0.07), and correlates with it at Pearson 0.94–0.99 across layers and backbones.

The analytical COR tracks the measured collapse threshold: it keeps the ranking, nearly equals the threshold for the manipulations that collapse, and stays highly correlated with it across layers and backbones.

CORNTF2FFSDF
Analytical0.0070.0690.1670.189
Empirical0.010.070.891.40

Lemma 1 gives a sufficient condition for stable descent, ρ<∥∇Lt∥Lt,\rho<\frac{\|\nabla\mathcal{L}_t\|}{L_t}, so the analytical COR is a conservative lower bound. NT and F2F collapse at the standard radius, and for them the two values nearly coincide. This matches Appendix B.3, where the bound is tight in the collapse regime (COR < 0.05). The gap appears only for FS and DF, whose COR lies far above the collapse regime.

Across layers and backbones, the analytical and empirical COR of the four manipulations remain highly correlated:

SettingPearson
CLIP-L, layer 50.97
CLIP-L, layer 150.96
CLIP-L, last layer0.94
DINOv3-L, last layer0.99
SigLIP2-L, last layer0.98

Q3-1: Does test-time SBI use extra information, and is it reproducible?

Answer: SBI uses only the test frame itself and a public landmark detector. No label, paired real image, or mask is involved.

At inference, SBI uses only the current test frame and a public landmark detector, so it adds no information beyond the test input. Following SBI [2] (Shiohara & Yamasaki, CVPR 2022), we align faces with dlib's public 81-point landmarks (Section 5.1), copy the frame, apply appearance and geometric transforms, and blend the copy back with a soft mask from the landmark convex hull. No label, paired real image, annotation, or ground-truth mask is needed. The step-by-step procedure is given in Appendix D.1.3 for reproducibility.

Q3-2: Why do the CLIP-L UniFD results differ between Tables 2 and 8?

Answer: Table 8 (Table 12 in the revised version) kept early-stage numbers by mistake. Table 2 is final and Table 8 has been corrected. After correction, CoRIT still outperforms UniFD on CLIP-L by 13.7 points.

References

[1] Jinlong Liu, Yunzhi Bai, Guoqing Jiang, Ting Chen, and Huayan Wang. Understanding Why Neural Networks Generalize Well Through GSNR of Parameters. In ICLR, 2020.

[2] Kaede Shiohara and Toshihiko Yamasaki. Detecting Deepfakes with Self-Blended Images. In CVPR, pp. 18699–18708, 2022.

Reviewer_ysaeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the generalization limitations of frozen vision foundation models for high-fidelity face forgery detection. Based on the mentioned observation, the paper introduces the Critical Optimization Radius (COR) as a measure of optimization stability and relates it to the local gradient norm, the largest Hessian eigenvalue, and the Gradient Signal-to-Noise Ratio (GSNR). The experimental results have proven its effectiveness.

Critical strengths
  1. High-fidelity manipulations increasingly contain only weak forensic traces. Studying whether frozen VFM features preserve such traces is timely and meaningful.
  2. The optimization-collapse observation is interesting. The substantially different SAM failure radii across manipulation types provide a potentially useful perspective on the interaction among feature quality, data difficulty, and classifier optimization.
  3. CoRIT improves both cross-manipulation and cross-dataset performance.
Critical weaknesses
  1. COR is sensitive to the training trajectory and parameterization. The proposed definition seems to depend on training duration, early stopping, learning rate, feature normalization, loss scaling, and classifier parameterization. This weakens the claim that COR directly measures an intrinsic property of the frozen representation.
  2. The causal link between fidelity and collapse is under-supported. The cross-manipulation analysis contains only four points and conflates fidelity with manipulation method and artifact type. FID is also a distribution-level measure and may not directly represent forensic difficulty.
  3. Some gains may arise from additional views or capacity rather than the claimed mechanism. CoRIT uses both the original and SBI inputs, additional region tokens, and intermediate-layer fusion. The present ablations do not exclude simpler explanations such as two-view augmentation, additional token capacity, or generic multi-layer fusion.
Questions for authors
  1. What is the correlation between analytical and empirical COR across manipulations, layers, and backbones?
  2. How sensitive is COR to feature normalization, classifier initialization, loss scaling, optimizer choice, and learning rate?
  3. How are SBI and the three spatial regions produced at inference time? Are paired real images, human annotations, or ground-truth manipulation masks required?
3 comments

We sincerely thank the reviewer for the constructive feedback. Below, we respond point by point. Each point starts with a short answer, followed by details. We hope these responses address your concerns.

W1 & Q2: Does COR depend on learning rate, feature normalization, loss scaling, and other training settings?

Answer: The absolute value of COR depends on the training setting, but the relative COR within each setting stays stable across all tested normalizations, learning rates, feature scales, and optimizers. This does not affect the original conclusions of our paper.

COR inherits the setting dependence of SAM's radius ρ, but its comparisons are stable. COR is the critical value of ρ, a radius in parameter space, and ρ itself must be retuned when the learning rate or feature scale changes. GSNR is also a training-time quantity that depends on the optimizer state.

Feature normalization barely changes the absolute COR:

NormNTF2FFSDF
Raw0.010.070.891.40
L20.010.121.261.35
RMS0.0080.061.031.14
Δ_norm/Δ_forgery0.1%4.3%27%19%

The last row divides the largest change of each manipulation across the three normalizations by the COR gap across manipulations under Raw. Normalization shifts each manipulation by at most 27% of this gap and by 13% on average. These shifts are small relative to the gaps between manipulations, and the ranking NT < F2F < FS < DF is preserved under all three normalizations.

Learning rate, feature scale, and optimizer change the absolute COR, but each change moves all four manipulations in the same direction (no COR decreases as the scale grows):

lrNTF2FFSDF
5e-40.0040.050.541.24
1e-30.010.070.891.40
2e-30.0140.121.491.67
Feature scaleNTF2FFSDF
×0.250.0020.060.770.85
×0.50.0080.060.830.96
×10.010.070.891.40
×20.010.131.321.62
×40.030.281.832.57
Optimizer (CLIP-ViT-L)NTF2FFSDF
SAM-SGD0.010.070.891.40
SAM-AdamW0.040.091.232.08

We therefore also report the relative COR Zd=CORd−COR‾sCOR,Z_d=\frac{\mathrm{COR}_d-\overline{\mathrm{COR}}}{s_{\mathrm{COR}}}, where COR‾\overline{\mathrm{COR}} and sCORs_{\mathrm{COR}} are the mean and standard deviation of COR over the four manipulations under the same setting. The relative COR stays stable:

lrNTF2FFSDF
5e-4-0.91-0.820.161.57
1e-3-1.00-0.900.511.39
2e-3-1.06-0.930.881.11
Δ_lr/Δ_forgery6.3%4.6%30%19%
Feature scaleNTF2FFSDF
×0.25-1.07-0.920.891.10
×0.5-1.05-0.930.841.14
×1-1.00-0.900.511.39
×2-1.07-0.900.781.20
×4-1.08-0.850.621.31
Δ_scale/Δ_forgery3.3%3.3%16%12%
Optimizer (CLIP-ViT-L)NTF2FFSDF
SAM-SGD-1.00-0.900.511.39
SAM-AdamW-0.96-0.910.441.44
Δ_opt/Δ_forgery1.7%0.4%2.9%2.1%

Every setting change moves the relative COR by at most 30% of the gap across manipulations and by 8% on average. Cross-setting effects are too small to reorder the manipulations, and the ranking NT < F2F < FS < DF holds in every setting. The conclusions of the paper are therefore unaffected.

Loss scaling does not change COR. Multiplying the loss by cc multiplies both ∣∇L∣\\|\nabla\mathcal{L}\\| and λmax⁡\lambda_{\max} by cc, so the analytical COR in Eq. (2) is unchanged. The SAM perturbation ρ,g/∣g∣\rho\\,g/\\|g\\| is also unchanged, so loss scaling acts only as a learning-rate change.

Training duration and early stopping do not drive our conclusions. Under prolonged training, baselines show U-shaped overfitting while CoRIT stays stable (Appendix D, Figure 8).

Our Main Position

Like GSNR [1] (Liu et al., ICLR 2020), COR is a training-time diagnostic: because the backbone is frozen, static diagnostics such as parameter norms are identical for every forgery type and cannot reveal how the same model generalizes on different data. These differences surface only during training, and COR exposes them, diagnosing why the same frozen model generalizes well on some forgery types but not on others. We also delineate the conditions under which our theory holds and verify them experimentally, and the original conclusions of the paper remain unchanged.

W2: Is the fidelity–collapse link causal, given only four manipulations and a distribution-level FID?

Answer: We address this in Section 3.2 by varying fidelity within a single manipulation (DF), keeping the method and artifact pattern fixed. COR and held-out AUC both decrease as fidelity increases.

Section 3.2 removes the confound of method and artifact type by varying fidelity within one manipulation. For each matched real–DF pair, we construct xα=xreal+α(xfake−xreal)x_\alpha=x_{\mathrm{real}}+\alpha(x_{\mathrm{fake}}-x_{\mathrm{real}}) (Section 3.2, Appendix E.2). This keeps the DF method, the frame, and the spatial pattern of the artifact fixed, and only scales the artifact magnitude. As α\alpha decreases (higher fidelity), COR and held-out AUC consistently decrease (Figure 3(c)).

Beyond FID, we also use a per-pair measure. The semantic displacement is computed per matched real–fake pair on 4,480 pairs and gives the same ordering (Table 19: NT 0.046 < F2F 0.062 < FS 0.103 < DF 0.115). The FID ordering also holds for two sample sizes (Table 18).

W3: Could the gains come from extra views, extra tokens, or generic multi-layer fusion?

Answer: Using SBI as data augmentation adds only 0.3 (78.9 vs. 78.6) and extra tokens alone add only 0.6. The gain of multi-layer fusion follows COR: fusing layer 18 (COR 2.47) gives 88.9, while layer 5 (COR 0.42) gives 86.2.

Each simpler explanation gives far less than CoRIT. Same setting as Table 3 (average AUC over CDFv2, DFDC, facedancer, fsgan):

ConfigurationAvg. AUC (%)
UniFD (baseline)78.6
Original + SBI as data augmentation78.9
Extra region tokens only (capacity)79.2
HRI only (multi-layer fusion, Table 3)82.3
CoRIT (full)88.8

Using SBI as data augmentation without CGP guidance and adding extra token capacity each barely help. With the same number of region tokens, random regions reach only 80.7 vs. 88.9 (Table 11).

The gain of multi-layer fusion also follows COR. The fusion layer with the highest COR gives the best AUC:

Fusion layerAvg. AUC (%)COR
586.20.42
1888.92.47
2088.20.40

Q1: How well does the analytical COR match the empirical COR across manipulations, layers, and backbones?

Answer: The analytical COR keeps the empirical ranking on all four manipulations, nearly equals it in the collapse regime (NT: 0.007 vs. 0.01, F2F: 0.069 vs. 0.07), and correlates with it at Pearson 0.94–0.99 across layers and backbones.

The analytical COR tracks the measured collapse threshold: it keeps the ranking, nearly equals the threshold for the manipulations that collapse, and stays highly correlated with it across layers and backbones.

CORNTF2FFSDF
Analytical0.0070.0690.1670.189
Empirical0.010.070.891.40

Lemma 1 gives a sufficient condition for stable descent, ρ<∥∇Lt∥Lt,\rho<\frac{\lVert\nabla\mathcal{L}_t\rVert}{L_t}, so the analytical COR is a conservative lower bound. NT and F2F collapse at the standard radius, and for them the two values nearly coincide. This matches Appendix B.3, where the bound is tight in the collapse regime (COR < 0.05). The gap appears only for FS and DF, whose COR lies far above the collapse regime.

Across layers and backbones, the analytical and empirical COR of the four manipulations remain highly correlated:

SettingPearson
CLIP-L, layer 50.97
CLIP-L, layer 150.96
CLIP-L, last layer0.94
DINOv3-L, last layer0.99
SigLIP2-L, last layer0.98

Q3: Does inference need paired real images, annotations, or ground-truth masks?

Answer: None of them is needed. SBI and the three regions are generated from the test frame itself with the public dlib landmark detector.

At inference, CoRIT needs only the current frame and a public landmark detector. Following SBI [2] (Shiohara & Yamasaki, CVPR 2022), we align faces with dlib's public 81-point landmark predictor (Section 5.1). The SBI view is made by copying the current frame, applying appearance and geometric transforms to the copy, and blending it back with a soft mask from the landmark convex hull. The three regions (face foreground, boundary near the face contour, and background) are built from the same landmarks and mapped to the ViT patch grid. RRM then refines token selection with the CGP. No paired real image, human region annotation, test label, or ground-truth manipulation mask is used. The manipulation masks shown in Appendix E only visualize the interpolation study.

References

[1] Jinlong Liu, Yunzhi Bai, Guoqing Jiang, Ting Chen, and Huayan Wang. Understanding Why Neural Networks Generalize Well Through GSNR of Parameters. In ICLR, 2020.

[2] Kaede Shiohara and Toshihiko Yamasaki. Detecting Deepfakes with Self-Blended Images. In CVPR, pp. 18699–18708, 2022.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.