- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Transfer from pretrained models is useful when training classifiers with limited data. However, it remains unclear whether adversarial robustness comes from robust pretraining or from robust fine-tuning on the downstream task. Prior studies have reached both conclusions, but they used different comparison settings. This study compares combinations of pretraining method, fine-tuning objective, and whether the feature extractor is frozen or all layers are updated. The authors show that robustness in a frozen feature extractor is largely inherited from pretraining, whereas robust fine-tuning can also confer robustness when all layers are updated. They also show that the advantage of robust pretraining grows as the amount of data on downstream tasks decreases. Finally, they find that "standard pretraining -> robust fine-tuning" and "robust pretraining -> robust fine-tuning" produce different internal representations, even when their accuracies are nearly identical.
Critical strengths
The study fills gaps in comparisons made by prior work and provides analyses that help build a more comprehensive understanding of robust pretraining and fine-tuning.
Critical weaknesses
-
The novelty of the study is insufficient. The main difference between this study and the two works, Vaishnavi et al. (2024) and Kopetzki et al. (2025), is its examination of settings between those studied previously, chiefly freezing the feature extractor or updating all layers. Bringing these settings together in a single paper is a contribution, but I question whether it is sufficient for acceptance. Moreover, Hua et al. (2024), Ngnawe et al. (2026), both cited in the paper, and the following study have conducted extensive experiments on this topic. A more detailed discussion of the study’s novelty is needed.
J. Zhang et al., ImageNet Pre-training also Transfers Non-Robustness, AAAI, 2023. -
The experiments are insufficient in several respects, some of which the authors acknowledge. First, the architectures studied are limited to ResNet and WideResNet. Given recent trends, Vision Transformers should also be included. On line 477, the authors state that a controlled comparison involving random initialization is difficult. However, a comparison is still possible with standard pretraining held as a common condition. I do not think the difficulty with random initialization justifies excluding Vision Transformers. In addition, experiments in the low-data regime are conducted only on CIFAR-10. This provides limited support for one of the study’s key conclusions: that robust pretraining becomes more advantageous as the amount of data on downstream tasks decreases. This finding should be verified on at least two additional datasets.
Questions for authors
I would like the authors to reconsider the two weaknesses above, particularly the novelty of this study.
4 comments
Thank you for the authors' response. The changes in the revised PDF are difficult to identify. It would be helpful if the authors could highlight the revisions, for example by using a different text color, as many authors do.
Thank you for the suggestion. We have uploaded a revised PDF in which all changes since the original submission are highlighted in yellow.
I thank the authors for their clarification. My assessment remains unchanged. As other reviewers have also noted, I do not believe the paper offers sufficient novelty to warrant acceptance. Fundamentally, this work seems to amount to conducting experiments under narrowly defined settings that prior work has not explored. I find it difficult to identify a substantial new contribution made by this paper. For example, while prior work may not have addressed deterministic certified robustness, it remains unclear what important new insights emerge from examining it. At least two of the three reviewers have expressed concerns about the paper’s novelty. If these concerns reflect a misunderstanding on our part, the authors should substantially restructure the paper so that readers can clearly understand its novel contributions.
We thank the reviewer for the detailed review and the concrete suggestions. We address the highlighted weaknesses as follows:
W1 We agree that our positioning against Hua et al. (2024), Ngnawé et al. (2026) and Zhang et al. (2023) was insufficient. We have adapted Table 1 to include these works and added an extended version in Appendix A that separates architectures and data scales. We have changed the wording of the introduction and related work to discuss each [L051, 069, 818]. This revised related work shows that none evaluate certified robustness for our threat model or vary data scale within a fixed domain. We have re-centred our contributions on these two points, and frame the adaptation-mechanism result as extending Hua et al. to CNNs with from-scratch controls [L078, 080, 105:117].
W2a We agree that the absence of a random-initialisation control is not a sufficient reason to exclude ViTs from the evaluation, and that a comparison holding standard pre-training fixed is possible. Having revisited the literature, we revised the reasoning [L527]. Empirical $ \ell_{\infty} $ transfer for ViT and Swin across adaptation mechanisms is already covered by Hua et al. and Ngnawé et al., so we do not pursue that direction. Instead, our CNN results complement their ViT results and agree with them in direction. What distinguishes our study is the certified track; to our knowledge, deterministic certified training and verification do not currently scale to vision transformers at this size, which is why we exclude ViTs in this work. We now state this in Section 5 [L529].
W2b We agree that a single target limits the generality of this conclusion. We chose a synthetic reduction because it holds the domain fixed in a controllable way, with the naturally small targets having somewhat similar per-class sample sizes (though these vary widely, which is why we used a synthetic reduction). We now scope the data-scale claims to acknowledge this limitation [L523].