acceptodds
Under review as a conference paper at ICLR 2027

Empirical and Certified Robustness Across Pre-training Regimes in Transfer Learning

Abstract

Transfer learning is a natural fit for safety-critical applications, where task-specific data is scarce but models have to be robust against adversarial perturbations. Robustness can be instilled during pre-training, during fine-tuning, or both, where robust pre-training is by far the most expensive of these. Yet it remains unclear when this expensive step is actually needed: prior work reaches opposing conclusions, and each study fixes a single adaptation mechanism while letting downstream data scale vary together with the target domain. In this paper, we take on a new comparative perspective to investigate empirical and certified robustness across pre-training regimes, fine-tuning objectives, adaptation mechanisms, and downstream data scales, under different adversary norms, also exposing trade-offs. Our findings show that the adaptation mechanism decides the answer, also reconciling prior work's settings and conclusions: With a frozen backbone, robustness can only be inherited from pre-training, while with an updateable backbone, robust fine-tuning instils robustness and standard fine-tuning largely erases it. We further show that robust pre-training nonetheless gains value for empirical robustness as downstream data becomes scarce. Finally, a layer-wise representational analysis reveals that comparable robustness can arise from distinct internal representations.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
adversarial robustness, transfer learning, certified robustness, fine-tuning, robust pre-training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LsJeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Transfer from pretrained models is useful when training classifiers with limited data. However, it remains unclear whether adversarial robustness comes from robust pretraining or from robust fine-tuning on the downstream task. Prior studies have reached both conclusions, but they used different comparison settings. This study compares combinations of pretraining method, fine-tuning objective, and whether the feature extractor is frozen or all layers are updated. The authors show that robustness in a frozen feature extractor is largely inherited from pretraining, whereas robust fine-tuning can also confer robustness when all layers are updated. They also show that the advantage of robust pretraining grows as the amount of data on downstream tasks decreases. Finally, they find that "standard pretraining -> robust fine-tuning" and "robust pretraining -> robust fine-tuning" produce different internal representations, even when their accuracies are nearly identical.

Critical strengths

The study fills gaps in comparisons made by prior work and provides analyses that help build a more comprehensive understanding of robust pretraining and fine-tuning.

Critical weaknesses
  1. The novelty of the study is insufficient. The main difference between this study and the two works, Vaishnavi et al. (2024) and Kopetzki et al. (2025), is its examination of settings between those studied previously, chiefly freezing the feature extractor or updating all layers. Bringing these settings together in a single paper is a contribution, but I question whether it is sufficient for acceptance. Moreover, Hua et al. (2024), Ngnawe et al. (2026), both cited in the paper, and the following study have conducted extensive experiments on this topic. A more detailed discussion of the study’s novelty is needed.
    J. Zhang et al., ImageNet Pre-training also Transfers Non-Robustness, AAAI, 2023.

  2. The experiments are insufficient in several respects, some of which the authors acknowledge. First, the architectures studied are limited to ResNet and WideResNet. Given recent trends, Vision Transformers should also be included. On line 477, the authors state that a controlled comparison involving random initialization is difficult. However, a comparison is still possible with standard pretraining held as a common condition. I do not think the difficulty with random initialization justifies excluding Vision Transformers. In addition, experiments in the low-data regime are conducted only on CIFAR-10. This provides limited support for one of the study’s key conclusions: that robust pretraining becomes more advantageous as the amount of data on downstream tasks decreases. This finding should be verified on at least two additional datasets.

Questions for authors

I would like the authors to reconsider the two weaknesses above, particularly the novelty of this study.

4 comments

We thank the reviewer for the detailed review and the concrete suggestions. We address the highlighted weaknesses as follows:

W1 We agree that our positioning against Hua et al. (2024), Ngnawé et al. (2026) and Zhang et al. (2023) was insufficient. We have adapted Table 1 to include these works and added an extended version in Appendix A that separates architectures and data scales. We have changed the wording of the introduction and related work to discuss each [L051, 069, 818]. This revised related work shows that none evaluate certified robustness for our threat model or vary data scale within a fixed domain. We have re-centred our contributions on these two points, and frame the adaptation-mechanism result as extending Hua et al. to CNNs with from-scratch controls [L078, 080, 105:117].

W2a We agree that the absence of a random-initialisation control is not a sufficient reason to exclude ViTs from the evaluation, and that a comparison holding standard pre-training fixed is possible. Having revisited the literature, we revised the reasoning [L527]. Empirical $ \ell_{\infty} $ transfer for ViT and Swin across adaptation mechanisms is already covered by Hua et al. and Ngnawé et al., so we do not pursue that direction. Instead, our CNN results complement their ViT results and agree with them in direction. What distinguishes our study is the certified track; to our knowledge, deterministic certified training and verification do not currently scale to vision transformers at this size, which is why we exclude ViTs in this work. We now state this in Section 5 [L529].

W2b We agree that a single target limits the generality of this conclusion. We chose a synthetic reduction because it holds the domain fixed in a controllable way, with the naturally small targets having somewhat similar per-class sample sizes (though these vary widely, which is why we used a synthetic reduction). We now scope the data-scale claims to acknowledge this limitation [L523].

Thank you for the authors' response. The changes in the revised PDF are difficult to identify. It would be helpful if the authors could highlight the revisions, for example by using a different text color, as many authors do.

Thank you for the suggestion. We have uploaded a revised PDF in which all changes since the original submission are highlighted in yellow.

I thank the authors for their clarification. My assessment remains unchanged. As other reviewers have also noted, I do not believe the paper offers sufficient novelty to warrant acceptance. Fundamentally, this work seems to amount to conducting experiments under narrowly defined settings that prior work has not explored. I find it difficult to identify a substantial new contribution made by this paper. For example, while prior work may not have addressed deterministic certified robustness, it remains unclear what important new insights emerge from examining it. At least two of the three reviewers have expressed concerns about the paper’s novelty. If these concerns reflect a misunderstanding on our part, the authors should substantially restructure the paper so that readers can clearly understand its novel contributions.

Reviewer_RqYsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper systematically investigates the role of robust pre-training in adversarial transfer learning. The authors compare different pre-training regimes, fine-tuning objectives, and adaptation mechanisms under both empirical and certified robustness. They find that linear probing largely preserves robustness inherited from pre-training, while full fine-tuning can either improve or erase robustness depending on the training objective. They also show that robust pre-training becomes more important when downstream data is limited, and that models with similar robustness can have different internal representations.

Critical strengths

The paper provides a systematic comparison of different pre-training and fine-tuning strategies for adversarial robustness. The paper is easy to follow.

Critical weaknesses

This paper is mainly an empirical study without proposing a new method. However, I am concerned about the novelty of its findings. First, the conclusions in RQ1 (Adaptation Mechanisms) have already been largely explored in prior work [1]. Second, the finding in RQ2 seems somewhat intuitive, as pre-training is generally expected to become more important when downstream training data is limited. Given these concerns, I feel the contributions of this paper are somewhat limited and fall below the bar for acceptance.

[1]: Hua, Andong, et al. "Initialization matters for adversarial transfer learning." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

Questions for authors

NA

2 comments

We thank the reviewer for their review and for pointing us to Hua et al. (2024), which we had not discussed adequately. We address their two weaknesses as follows:

W1 We agree that Hua et al. already compare linear probing with full and parameter-efficient adversarial fine-tuning, from standard and robust backbones, under an $ \ell_{\infty} $ adversary for Vision Transformers. We have corrected Table 1, the introduction and the contributions to credit this [L108]. Our RQ1 results go beyond Hua et al. in three respects, which we now state explicitly:

  1. We evaluate certified robustness, whereas Hua et al. evaluate empirical robustness only. To our knowledge, ours is the first study of robustness transfer under deterministic certification, and we find that the frozen-versus-updated pattern also holds there.
  2. Hua et al. study ViT and Swin without a training-from-scratch reference; we study CNNs with such controls, which shows, for example, that standard pre-training adds no robustness over training from scratch at reduced data scales.
  3. We also find a discrepancy, as Hua et al. report robustness after standard full fine-tuning of a robust backbone, whereas we find near-complete erasure on four of five targets under AutoAttack. We now highlight and discuss this more explicitly and address possible causes [L371].

W2 We agree that pre-training is expected to matter more when data is scarce. However, our findings reveal several nuances that motivate the study. (1) At reduced data, robust fine-tuning of a standard backbone does no better than training from scratch on every CIFAR-10 reduction and on both tracks. This contradicts Vaishnavi et al. (2024), who attribute the benefit to the general utility of pre-training. (2) For certified robustness, the effect nearly disappears, with the gap within one standard deviation at 5,000 and 500 examples per class. Prior work could not isolate either effect, because data scale varied with the target domain. We have sharpened contribution 3 to make this explicit [L113].

Thank you for the rebuttal. I appreciate the authors' acknowledgment of the overlap with prior work. In this case, the overall contribution is insufficient for acceptance and I will maintain my original score.

Reviewer_RtxfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when robust pre-training is needed for downstream robustness, focusing on interactions between adaptation mechanisms, training objectives, and downstream data availability. It compares no pre-training, standard pre-training, and robust pre-training, combined with standard or robust training objectives. Adaptation is performed through either linear probing with a frozen backbone or full-network fine-tuning; the no-pre-training controls are evaluated only through full-network training.

Within the evaluated settings, robust head training does not successfully establish robustness on frozen standard backbones, whereas frozen robust backbones retain substantial robustness. Full-network robust fine-tuning often establishes robustness, while standard fine-tuning substantially reduces empirical robustness or IBP-certified accuracy. The advantage of robust over standard pre-training increases as CIFAR-10 data availability decreases in the empirical track. In the certified track, training from scratch is competitive. A CKA analysis further shows that similar certified accuracies can arise from different deep representations.

Critical strengths
  1. The study jointly examines adaptation mechanisms and downstream data availability, producing useful evidence for practical transfer decisions. The paper explicitly distinguishes linear probing from full fine-tuning. It also changes the amount of CIFAR-10 training data while holding the target task and test set fixed, which is more informative than attributing differences across unrelated datasets to data scarcity.

  2. The deterministic certification track and training-from-scratch controls provide a valuable complement to attack-based evaluation.

Critical weaknesses

1. Insufficient evidence for the necessity of robust pre-training.

  • The fixed-ϵ\epsilon fine-tuning strategy may cause optimization failures for standard pre-trained models, making the comparison potentially unfair without an ϵ\epsilon-scheduling baseline.
  • The claim that robust pre-training becomes necessary in low-data regimes is overstated, since training from scratch still achieves 18.83% AutoAttack accuracy on CIFAR-10 1%.

2. Confounded experimental comparisons and overstated conclusions.

  • The empirical and certified robustness experiments use different architectures, datasets, resolutions, and perturbation budgets, preventing conclusions about the effects of robustness type alone.
  • Zero IBP-certified accuracy does not imply zero actual robustness, so claims that standard fine-tuning eliminates robustness are insufficiently supported.
Questions for authors
  1. Fine-tuning strategy: Does the advantage of robust pre-training persist when ϵ\epsilon-scheduling is applied to standard pre-trained models, particularly in low-data settings?

  2. Necessity of robust pre-training: How do the authors justify calling robust pre-training "necessary" when training from scratch still achieves nontrivial adversarial robustness?

  3. Experimental validity: Can the authors provide controlled comparisons between empirical and certified robustness, and attack-based evaluations for models with zero certified accuracy, to substantiate their conclusions?

1 comment

We thank the reviewer for their careful and positive review. We agree with their points and address their questions in the following way:

Q1 We did not run ϵ\epsilon -scheduling, so we cannot claim that the advantage of robust pre-training persists under it. Given Ngnawé et al. (2026), we expect the gap between Robust Adaptation and Fully Robust to shrink, especially at low data, where we observe exactly the fixed- ϵ\epsilon failure they describe. We now state that our reported gaps may overestimate the benefit of robust pre-training under scheduled recipes, and list combining scheduling with our design as the most direct follow-up [L409, 446].

Q2 We agree that the claim of necessity is not strictly true here, given the robustness recovered when training from scratch. We introduce a new label in Table 1, "helpful but not necessary", use it for this cell, and revise the RQ2 answer, the contributions and the conclusion accordingly [L054].

Q3a We agree that a fully controlled comparison would require certified training at ImageNet scale and 224-pixel resolution. We aimed to make the comparison as close as possible (much closer than prior work), with both tracks at an operational scale. We now restrict cross-track statements to the direction of effects, replace phrasing that attributes differences to the robustness type with track-level phrasing, and name the weaker certified backbone as a further confounder [L353].

Q3b We agree that IBP provides a loose lower bound. We now describe these results as the loss of IBP-certifiable robustness [L394, 332].

Comments on the paper

1 comment

We thank all three reviewers for their careful reading and constructive feedback. We see two shared concerns: positioning against prior work, specifically Hua et al. (2024), Ngnawé et al. (2026) and Zhang et al. (2023), and the strength of several claims. We have addressed these shared points as follows:

  1. Table 1 now credits Hua et al. for the $ \ell_{\infty} $ empirical adaptation mechanism comparison on vision transformers (ViTs). The introduction and related work discuss all three works directly.
  2. We re-centre the contribution on what no prior work covers: deterministic certified robustness evaluated alongside empirical robustness in one design, with data scale varied at a fixed target domain. The adaptation-mechanism finding is now framed as an extension of prior ViT results to CNNs with from-scratch controls.
  3. We qualify the low-data “necessity” claim (fixed-ϵ\epsilon recipe, non-trivial from-scratch robustness), restrict cross-track statements to the direction of observed effects, and describe zero IBP accuracy as a loss of certifiable, not of all, robustness.

Individual responses below reference the specific changes. We thank the reviewers again for feedback that has improved the work, and we welcome any further questions during the remaining time of the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.