- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the sensitivity of linear probes on frozen vision foundation models to adversarial parameter perturbations during SAM training. It introduces the Critical Optimization Radius (COR), relates an analytical approximation to the gradient signal-to-noise ratio (GSNR), and interprets layer-wise changes as evidence of attenuated forensic cues. The proposed CoRIT uses original–SBI feature differences to refine regional token selection, injects pooled features into auxiliary tokens, and combines intermediate and final-layer representations. Experiments evaluate cross-manipulation and cross-dataset detection while training only a lightweight classification head.
Critical strengths
-
The observed variation in SAM sensitivity across manipulation types, backbone layers, and model families is interesting. The radius sweeps and layer-wise analyses provide a concrete empirical phenomenon that merits investigation as a possible diagnostic of forensic representations.
-
CoRIT achieves meaningful gains with a frozen backbone. Component ablations support the usefulness of regional injection, refinement, and intermediate-layer fusion, while head-only training offers practical memory savings.
Critical weaknesses
-
The central interpretation of COR as a diagnostic of transferable forensic information is insufficiently established. Lemma 1 supplies a sufficient descent condition; exceeding its bound does not imply collapse or chance-level classification. Appendix B.4 acknowledges this distinction, but the subsequent representation-level conclusions require stronger validation of the analytical approximation against the observed collapse threshold. COR also depends on parameterization: rescaling features and inversely rescaling a linear head preserves predictions while changing the effect of a fixed parameter-space perturbation. The reported layer-wise trends therefore do not adequately separate loss of forensic information from scale and conditioning effects. The COR–GSNR theorem is a decomposition of the defined quantities, whose diagnostic interpretation additionally depends on the statistical-geometric factor remaining comparable across settings. These gaps directly affect the paper's principal mechanistic contribution, beyond the empirical usefulness of CoRIT.
-
The connection between the SBI-based gradient proxy and the gradients relevant to the actual detector needs stronger validation. Appendix C's Taylor expansion and Cauchy–Schwarz argument identify favorable alignment, but do not establish that observed SBI residuals align with task gradients. Its Jacobian argument concerns hypothetical backbone gradients, whereas CoRIT optimizes only the classification head. Consequently, reduced regional proxy variance does not by itself establish improved parameter-gradient GSNR. The component ablations provide encouraging evidence that the design is useful, but direct measurements are needed to distinguish the proposed gradient-signal explanation from the benefits of regional aggregation and multi-layer feature access.
Questions for authors
-
Assessment-critical validation: After controlling for feature scale and selecting optimizer settings on validation data, does COR retain predictive value for cross-domain AUC beyond ordinary linear-probe performance? How closely does the analytical approximation track the empirical collapse threshold?
-
Mechanism validation: Can the authors measure alignment between the SBI residual and the relevant feature gradient, and connect regional pooling to changes in the classifier-head gradient GSNR actually used in training?
4 comments
W1 & Q1: Are COR differences and layer-wise trends only effects of feature scale and parameterization?
Answer: COR is a training-time diagnostic like GSNR: its absolute value depends on scale, but the ranking and relative values stay stable across all tested normalizations, learning rates, feature scales, and optimizers. Our conclusions therefore remain valid.
We acknowledge that COR is not reparameterization-invariant, but its relative values under a fixed setting are stable. Feature normalization barely changes the absolute COR:
| Norm | NT | F2F | FS | DF |
|---|---|---|---|---|
| Raw | 0.01 | 0.07 | 0.89 | 1.40 |
| L2 | 0.01 | 0.12 | 1.26 | 1.35 |
| RMS | 0.008 | 0.06 | 1.03 | 1.14 |
| Δ_norm/Δ_forgery | 0.1% | 4.3% | 27% | 19% |
The last row divides the largest change of each manipulation across the three normalizations by the COR gap across manipulations under Raw. Normalization shifts each manipulation by at most 27% of this gap and by 13% on average. These shifts are small relative to the gaps between manipulations, and the ranking NT < F2F < FS < DF is preserved under all three normalizations.
Learning rate, feature scale, and optimizer change the absolute COR, but each change moves all four manipulations in the same direction (no COR decreases as the scale grows):
| lr | NT | F2F | FS | DF |
|---|---|---|---|---|
| 5e-4 | 0.004 | 0.05 | 0.54 | 1.24 |
| 1e-3 | 0.01 | 0.07 | 0.89 | 1.40 |
| 2e-3 | 0.014 | 0.12 | 1.49 | 1.67 |
| Feature scale | NT | F2F | FS | DF |
|---|---|---|---|---|
| ×0.25 | 0.002 | 0.06 | 0.77 | 0.85 |
| ×0.5 | 0.008 | 0.06 | 0.83 | 0.96 |
| ×1 | 0.01 | 0.07 | 0.89 | 1.40 |
| ×2 | 0.01 | 0.13 | 1.32 | 1.62 |
| ×4 | 0.03 | 0.28 | 1.83 | 2.57 |
| Optimizer (CLIP-ViT-L) | NT | F2F | FS | DF |
|---|---|---|---|---|
| SAM-SGD | 0.01 | 0.07 | 0.89 | 1.40 |
| SAM-AdamW | 0.04 | 0.09 | 1.23 | 2.08 |
We therefore also report the relative COR where and are the mean and standard deviation of COR over the four manipulations under the same setting. The relative COR stays stable:
| lr | NT | F2F | FS | DF |
|---|---|---|---|---|
| 5e-4 | -0.91 | -0.82 | 0.16 | 1.57 |
| 1e-3 | -1.00 | -0.90 | 0.51 | 1.39 |
| 2e-3 | -1.06 | -0.93 | 0.88 | 1.11 |
| Δ_lr/Δ_forgery | 6.3% | 4.6% | 30% | 19% |
| Feature scale | NT | F2F | FS | DF |
|---|---|---|---|---|
| ×0.25 | -1.07 | -0.92 | 0.89 | 1.10 |
| ×0.5 | -1.05 | -0.93 | 0.84 | 1.14 |
| ×1 | -1.00 | -0.90 | 0.51 | 1.39 |
| ×2 | -1.07 | -0.90 | 0.78 | 1.20 |
| ×4 | -1.08 | -0.85 | 0.62 | 1.31 |
| Δ_scale/Δ_forgery | 3.3% | 3.3% | 16% | 12% |
| Optimizer (CLIP-ViT-L) | NT | F2F | FS | DF |
|---|---|---|---|---|
| SAM-SGD | -1.00 | -0.90 | 0.51 | 1.39 |
| SAM-AdamW | -0.96 | -0.91 | 0.44 | 1.44 |
| Δ_opt/Δ_forgery | 1.7% | 0.4% | 2.9% | 2.1% |
Every setting change moves the relative COR by at most 30% of the gap across manipulations and by 8% on average. Cross-setting effects are too small to reorder the manipulations, and the ranking NT < F2F < FS < DF holds in every setting. The conclusions of the paper are therefore unaffected.
Layer-wise trends are not a scale effect. All layer-wise COR values are measured by feeding each layer's features into the same head under the same setting. Rescaling moves all four manipulations in the same direction. Across layers, however, COR rises for DF and FS but falls for NT and F2F (Figure 1(b)). A common scale or conditioning change cannot produce opposite trends, so the attenuation reflects the loss of forensic information. Across layers, γ also varies far less than GSNR (see W2), so the layer-wise COR trend is driven mainly by GSNR.
Our Main Position
COR is a training-time diagnostic, like GSNR [1] (Liu et al., ICLR 2020), and it captures differences that static measures cannot. All manipulations share the same frozen backbone, so static measures of its weights, such as parameter norms, are identical for every forgery type and cannot separate them. COR and GSNR are measured while training on each dataset, so they reflect how well the features support learning on that data. The cost is that their absolute values depend on training hyperparameters. Under the same training setting, however, the ranking of the relative values across the four manipulations stays consistent, so the conclusions of the paper are unaffected.
Q1: With optimizer settings selected on validation data, does CoRIT still go beyond an ordinary linear probe?
Answer: Yes. When UniFD (a linear probe) and CoRIT each select their own optimizer setting and checkpoint on validation data, CoRIT still leads by 4.9 (cross-manipulation) and 10.5 (cross-dataset) AUC points.
We let UniFD and CoRIT each select its optimizer setting (plain SGD, plain AdamW, or SAM-SGD) and checkpoint on the source validation set. UniFD selects plain SGD, and CoRIT selects SAM-SGD with ρ = 0.07. Average AUC over the four training sources:
| Method | Cross-manipulation | Cross-dataset |
|---|---|---|
| UniFD | 81.9 | 78.4 |
| CoRIT | 86.8 | 88.9 |
COR also predicts where CoRIT helps most: in Table 1, the gains are largest on the low-COR manipulations F2F (+12.5) and NT (+8.2).
W2: Does the COR–GSNR decomposition carry diagnostic meaning only if γ stays comparable?
Answer: Empirically, γ is stable across training steps (Appendix B.5.1, Figure 7), and across manipulations, layers, and backbones √γ varies less than √GSNR (at most 36% vs. 146%). Moreover, the variation of γ does not make the rankings of COR and GSNR disagree.
Eq. (4) is an identity, so its value lies in attribution: it splits COR into gradient-signal quality (GSNR) and a noise-to-curvature scale (γ): Intuitively, in the physical picture behind momentum-based optimizers such as Adam [2] (Kingma & Ba, ICLR 2015), the gradient acts as a force that moves the parameters. Taking the mean gradient as a force , gradient noise as , and curvature as a stiffness :
- COR is the displacement driven by the true signal.
- is the displacement driven by noise.
- is their ratio.
SAM [3] (Foret et al., ICLR 2021) shows that solutions that withstand larger parameter perturbations tend to generalize better, so a larger COR indicates that the representation supports better generalization. is the displacement scale of gradient noise, which mainly comes from random sampling differences within a batch. With a large batch, this variance is relatively stable, so intuitively γ does not change the rankings of COR and GSNR.
We measured how much each factor varies.
- Across training steps: the numerator and denominator of γ change together, so γ stays far more stable than either term (Appendix B.5.1, Figure 7).
- Across manipulations, layers, and backbones: since COR , we compare the variation of with that of :
| Comparison | √GSNR | √γ | COR |
|---|---|---|---|
| Manipulations (NT, F2F, FS, DF) | 35% | 11% | 50% |
| Layers (5, 10, 15, 20, 24) | 62% | 22% | 98% |
| Backbones (CLIP, DINOv3, SigLIP2) | 146% | 36% | 234% |
- Ranking: the empirical COR, the analytical COR, and GSNR all follow NT < F2F < FS < DF.
In every comparison, √γ varies only about one quarter to one third as much as √GSNR, and GSNR accounts for 70–75% of the variation in COR. γ therefore affects COR but stays comparable across settings, and it does not alter the ranking in our measurements. The diagnostic reading of COR is carried mainly by GSNR.
W3 & Q2: Do SBI residuals align with task gradients, and does regional pooling improve the head-gradient GSNR used in training?
Answer: The SBI residual shows a fairly strong positive correlation with the feature gradient (Spearman 0.51–0.61 across layers, vs. ≈0 when the residual is shuffled or replaced by a random vector). The Jacobian argument in Appendix C concerns the classification head, and the full model has the highest head-gradient GSNR (0.065 vs. 0.046–0.050 without any one module).
W3-1 & Q2-1: SBI Residual vs. Feature Gradient
The SBI residual is aligned with the feature gradient at every layer, and random controls show no alignment. Spearman correlation between the activation strengths of the feature gradient and of (i) the SBI residual, (ii) the SBI residual randomly shuffled, and (iii) a random vector:
| Layer | SBI residual | Global shuffle | Random vector |
|---|---|---|---|
| 5 | 0.58 | -0.0004 | -0.0009 |
| 10 | 0.61 | +0.0006 | -0.0001 |
| 15 | 0.59 | -0.0010 | -0.0001 |
| 20 | 0.51 | -0.0003 | +0.0003 |
The moderate-to-strong correlation disappears once the spatial correspondence is broken, so it is not a chance effect.
W3-2 & Q2-2: Regional Pooling and Head-gradient GSNR
The Jacobian argument in Appendix C concerns the classification head, not hypothetical backbone gradients. When we measure COR/GSNR at layer , the layer- features are fed directly into the classification head without passing through later layers. Regional proxy variance thus approximates the variance of the head-parameter gradient.
We directly measured the GSNR of the classification-head gradient used in training (average over four manipulations):
| RSI | RRM | HRI | Head GSNR |
|---|---|---|---|
| ✓ | ✓ | 0.046 | |
| ✓ | ✓ | 0.047 | |
| ✓ | ✓ | 0.050 | |
| ✓ | ✓ | ✓ | 0.065 |
Removing any module lowers the head GSNR, which directly links the regional design to the gradient signal actually used in training, beyond the benefit of regional aggregation or multi-layer access alone.
References
[1] Jinlong Liu, Yunzhi Bai, Guoqing Jiang, Ting Chen, and Huayan Wang. Understanding Why Neural Networks Generalize Well Through GSNR of Parameters. In ICLR, 2020.
[2] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015.
[3] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-Aware Minimization for Efficiently Improving Generalization. In ICLR, 2021.
We sincerely thank the reviewer for the constructive validation requests. We have added discussions and experiments that clearly delineate the conditions under which our theory holds. Below, we respond point by point. Each point starts with a short answer, followed by details. We hope these responses address your concerns.
The analytical COR tracks the measured collapse threshold.
The analytical COR preserves the empirical ordering of the four manipulations and is closest to the empirical value where collapse occurs (NT: 0.007 vs. 0.01, F2F: 0.069 vs. 0.07). It is below the empirical COR for every manipulation, which is what the theory predicts. Lemma 1 gives a sufficient condition for stable descent, ρ<Lt∥∇Lt∥, so training is guaranteed to be stable below this bound and may remain stable above it.
NT and F2F collapse at the standard radius, and for them the two values nearly coincide. This matches Appendix B.3, where the bound is tight in the collapse regime (COR < 0.05). The gap appears only for FS and DF, whose COR lies far above the collapse regime.
Across layers and backbones, the analytical and empirical COR of the four manipulations remain highly correlated: