acceptodds
Withdrawn from ICLR 2027

Attacking a Linear Probe measures the Head as much as the Encoder

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Adversarial evaluations of pretrained vision encoders often freeze the encoder, train a linear classifier, and attack the resulting model. How much of the reported robustness belongs to the classifier head? Across seven ImageNet-1k probes, we retrain six heads without their BatchNorm layer while holding the encoder and seed fixed. Under the AutoAttack ensemble, this intervention raises MoCo v3's robust accuracy from 3.88% to 6.26%: a change equal to 30% of the original between-encoder spread. In all six pairs, the sign of the change matches whether the original BatchNorm layer's mean feature-wise scale is below one (attenuation) or above one (amplification), with three differences significant under corrected paired tests. Cross-entropy PGD exaggerates the effect enough to reverse one clear pairwise encoder ordering; the ensemble does not resolve a rank reversal. A second intervention isolates attack sensitivity: dividing a trained head's logits by a positive constant preserves its decision function, yet changes PGD accuracy by up to 28.7 percentage points on MoCo v3. The extreme increase reflects gradient masking, not improved robustness. The two trained heads also differ under an attack with a scale-invariant objective. By contrast, the BatchNorm intervention changes ImageNet-C accuracy by at most 2.0 percentage points, leaving the large gaps between encoders essentially unchanged. These results support controlling the head and checking the attack when using frozen probes to compare representations. They establish dependence on the trained probe, while leaving the roles of head scaling and optimisation only partly separated.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Adversarial Robustness, Adversarial Evaluation, Self-Supervised Learning, Representation Learning, Linear Probing, Vision Transformers

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_LkSNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how a classifier head influences the adversarial robustness of a pretrained vision encoder (kept frozen). They evaluate seven vision encoders with different pretraining objectives, comparing classifiers trained with and without BatchNorm while keeping the encoders frozen. They evaluate adversarial robustness using FGSM-20, PGD, and AutoAttack, and assess corruption robustness on ImageNet-C. They also investigate the effect of logit rescaling on adversarial evaluation.

Critical strengths

The study of the influence of linear probes on the adversarial robustness in the context of transfer learning from a pretrained vision encoder is an important problem.

Critical weaknesses

Weaknesses

1. Problem formulation and motivation

L038–L039: “how much can it change if the encoder stays fixed and only the head changes?” Hua et al. (2024) [1] already showed that the linear head matters. A more precise formulation of the research question is required.

L047: “A comparison assembled across such recipes varies the head alongside the encoder” Hua et al. (2024) [1] already evaluate different linear classifier heads on the same frozen encoder.

L048: “Our experiments measure the size of this confound in our setup; whether it explains disagreements across studies remains open.” Unclear. What does “size” mean here? What do you mean by “disagreements across studies”?

L050: “First, retraining a different head changes the classifier f = head ◦ encoder, even though the representation is fixed.” This is a truism. Why do you need to test it?

L052: “Second, an attack’s estimate can change even when the classifier’s decisions do not” It is not clear what this means, and the following sentence (L053) seems to contradict it.

2. Novelty and contributions

L080: Contribution 1 Predictable from Hua et al. (2024) [1], unless there is a specific aspect you want to study.

L081: Contribution 2 Scaling logits obfuscates gradients [2, 5]. AutoAttack [3] contains APGD with the Difference of Logits Ratio (DLR) loss, which is invariant to logit rescaling. For more on the proper evaluation of adversarial robustness, please read [5].

L084: Contribution 3 This is not a clearly stated contribution with an understandable, falsifiable claim.

3. Related work

L042: “Frozen-probe work reports objective- and architecture-dependent robustness rankings” The cited work is more about adversarial robustness in visual self-supervised representation systems than robustness evaluation in general.

L091: Related work on “Evaluating frozen representations” The content and citations in this paragraph are not coherent.

L098: Related work on “Self-supervision and robustness” This does not seem relevant.

4. Experimental evaluation and setup

L057 The evaluation at 1/255 is absolutely weak compared with the perturbation budgets used for most datasets in the adversarial robustness literature.

L119: “Models and heads” For backbones, please check [6] and [7].

References

[1] Hua et al. (2024). Initialization Matters for Adversarial Transfer Learning. CVPR 2024. https://arxiv.org/abs/2312.05716

[2] Carlini and Wagner (2017). Towards Evaluating the Robustness of Neural Networks. IEEE S&P 2017. https://arxiv.org/abs/1608.04644

[3] Croce and Hein (2020). Reliable Evaluation of Adversarial Robustness with an Ensemble of Diverse Parameter-Free Attacks. ICML 2020. https://arxiv.org/abs/2003.01690

[4] Athalye, Carlini, and Wagner (2018). Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples. ICML 2018. https://arxiv.org/abs/1802.00420

[5] Carlini et al. (2019). On Evaluating Adversarial Robustness. arXiv preprint. https://arxiv.org/abs/1902.06705

[6] Heuillet et al. (2025). A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy. NeurIPS 2025. https://arxiv.org/abs/2508.14079

[7] Goldblum et al. (2023). Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks. NeurIPS 2023 (Datasets and Benchmarks Track). https://arxiv.org/abs/2310.19909

Questions for authors

cf. weaknesses

Reviewer_q995OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how retraining classifier heads and rescaling logits affect attack-based robustness measurements of frozen vision encoders, contrasting these effects with corruption accuracy.

Critical strengths

Separating changes to the trained classifier from sensitivity of the attack implementation is a useful evaluation question, and the paired-image controls help expose this distinction.

Critical weaknesses
  1. The incremental contribution and central explanatory result are not sufficiently established. In lines 040–049 and 080–114, the paper connects probe choices, BatchNorm, and attack reliability, but does not convincingly establish what general insight goes beyond existing warnings about readout dependence and gradient masking. Overall the Introduction is not well organized. It is quiet confusing when there is whole figure in this part but only few line to explain the Fig. 1. And it will much clear if the contributions are shown as bullets. In the part of experiment, lines 253–254 acknowledge that head scaling, conditioning, and optimization remain entangled, and lines 305–307 limit the scale control to MoCo v3. Please make the main hypothesis and advance over prior work explicit, and organize the evidence around controlled comparisons that separate these mechanisms. I noticed that there is no formal theoretical analysis. A formal theorem is not essential, but the empirical contribution needs a stronger explanatory or methodological conclusion.

  2. The evidence and presentation do not yet support a general statement about encoder comparisons. The abstract (lines 013–030) combines head retraining, scale-sensitive attacks, and corruption results without making their distinct implications sufficiently accessible. More importantly, every head has only one training seed (lines 191–198, 414–421). Please report repeated head training with uncertainty on the differences, match or vary optimization outcomes, and distinguish robust empirical conclusions from illustrative diagnostics.

Overall, my main concern is the presentation. The paper is difficult to follow, and its eight sections lack a coherent structure. More importantly, the work appears entirely empirical, without even a general formulation of the problem, which further obscures its motivation and scope. Substantial restructuring and clearer problem formulation are needed to make the contribution understandable.

Questions for authors
  1. Motivation and novelty What is the principal new conclusion beyond the established dependence on readout choice and attack reliability? And please define “probing recipes” when first introduced, state a concrete hypothesis, and provide a structured comparison with the closest evaluation studies explaining what the present experiments add.
  2. Mechanistic separation Since evaluation-mode BatchNorm can be folded into an affine head, how much of the observed difference comes from the effective classifier learned through optimization rather than normalization or scaling itself? Please add function-preserving controls and matched-convergence retraining, and extend the scale controls beyond MoCo v3 if a broader explanation is intended.
  3. Repeatability and interpretation Do the head effects and encoder orderings persist across training seeds? Please report uncertainty over repeated training as well as the existing paired-image tests, and explain what conclusions remain under the stronger attack when a clear rank reversal is not observed.
Reviewer_9Bp7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The robustness of pretrained vision models, such as MoCo and DINOv2, is often evaluated by freezing the backbone, training a linear classifier on ImageNet, and attacking the resulting model. Although these evaluations are used to compare backbone robustness, the classifier head’s influence may not be adequately considered. The authors investigate how the measured robustness of various backbones could be affected by the classification head, specifically by the absence of BatchNorm. They conduct experiments on six models and find that attack success rates change for most models in the absence of BatchNorm under AutoAttack and PGD. This indicates that the classification head needs to be considered when evaluating image backbone robustness.

Critical strengths
  • The authors find that the classification head needs to be considered when evaluating image encoder robustness, connected to the confound in robustness comparisons between encoders.
Critical weaknesses
  1. The contribution is limited. Showing that the absence of BN changes measured robustness and that the head needs to be considered in robustness comparisons is useful, but provides limited additional insight. The contribution could be strengthened by exploring which backbones are more sensitive to head choices, why this sensitivity differs, or how image encoder robustness should be compared (see question 1).

  2. The accuracy-robustness trade-off is insufficiently analyzed. The significant robustness improvements for MoCo v3 and MAE are accompanied by clean accuracy drops. These results may be associated with the accuracy-robustness trade-off [1]. The authors need to discuss this when analyzing the experimental results.

[1] Robustness May Be at Odds with Accuracy, ICLR 2019

Questions for authors
  1. It would be better to explore which architectures are more sensitive to head choices and what properties explain this sensitivity. Measuring accuracy and robustness under various classification-head settings (beyond the absence of BN) and training configurations (change learning rates, training seeds, etc.), and observing how the trends depend on architectures, could reveal new insights into these architectures and strengthen the contribution.

  2. Generalization beyond ImageNet. Evaluating additional downstream datasets would help determine whether head sensitivity depends on the classification task.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.