- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates adversarial robustness in downstream adaptation of pretrained Vision Transformers, termed ViT, using parameter-efficient fine-tuning, termed PEFT. The authors state that standard adaptation lacks robustness against adversarial examples. They also note that standard adversarial training, termed AT, cannot balance clean accuracy and robustness under a limited parameter budget. To address this issue, the authors propose Smoothness and Token Alignment Regularization, termed STAR. The method decomposes the adversarial objective into three components. These components include a class-token cosine alignment loss at the representation level, an interpolated token smoothness loss at the distribution level, and weighted cross-entropy losses at the decision level. The authors introduce two hyperparameter configurations named STAR-RE and STAR-GE. They claim that this approach achieves a better trade-off between clean accuracy and robustness under constrained parameter budgets.
Critical strengths
-
Comprehensive experimental evaluation across diverse settings: The paper provides an extensive empirical evaluation. The authors test six downstream adaptation methods, including LP, PT, LoRA, PFT, Adapter, and FFT, alongside five training conditions across three datasets. The evaluations cover six ViT-specific black-box transfer attacks as well as several white-box attacks. In addition, the appendix includes adaptation experiments on a self-supervised DINOv2 backbone.
-
Clear practical relevance of the problem setting: The paper addresses a common deployment scenario where practitioners adapt a shared pretrained backbone using parameter-efficient fine-tuning. It explores the security impact of restricted parameter updates. This investigation clarifies the need for robust adaptation under fixed parameter budget constraints.
Critical weaknesses
-
Limited theoretical novelty and packaging of regular hyperparameters as distinct modes: The components of the proposed loss function are combinations of existing techniques. These include cosine distance alignment in feature space, KL divergence consistency on interpolated features, and standard cross-entropy. The method does not incorporate specialized architectural designs for the attention mechanism or patch structure of ViTs. Furthermore, the two modes highlighted in the main text, named STAR-RE and STAR-GE, are merely two manually selected sets of loss weights detailed in Appendix C.4. This reflects standard hyperparameter tuning rather than algorithmic innovation.
-
Empirical results do not adequately demonstrate the necessity or efficacy of the proposed losses: The empirical data indicates that the proposed regularizations do not resolve the trade-off between clean accuracy and robustness. Under STAR-RE, clean accuracy drops substantially on CIFAR-10, falling from 98.37% to 37.56% under PT, and from 98.80% to 47.88% under PFT. Under STAR-GE, a large discrepancy appears between white-box PGD and standard AutoAttack evaluations. Multiple PEFT configurations drop to 0.00% robust accuracy under AutoAttack, indicating gradient masking. Overall, the experiments do not sufficiently prove that the proposed losses are necessary to establish genuine robustness.
Questions for authors
-
Selection mechanism of hyperparameter modes and severe drops in clean accuracy: The main text describes STAR-RE and STAR-GE as two adaptation modes with different emphases. However, Appendix C.4 reveals that both modes essentially correspond to two fixed sets of loss weights. Based on Table 1, STAR-RE causes substantial drops in clean accuracy under certain PEFT configurations. For example, on CIFAR-10, clean accuracy drops by over 60 percentage points under PT, and by over 50 percentage points under PFT. Meanwhile, STAR-GE provides relatively limited robustness gains under certain evaluated attacks. In real-world downstream tasks without prior knowledge of attack types, what general rule should practitioners follow to determine the specific weights of these four loss terms? Is there a single, stable weighting configuration that achieves consistent robustness gains without severely degrading standard classification accuracy?
-
Underlying causes of performance collapse under specific fine-tuning settings and justification for data exclusion: The paper argues that the proposed method is independent of specific parameter adaptation mechanisms and applies generally to diverse adaptation structures from LP to FFT. However, Appendix E.1 and Table 6 show that when using Adapter fine-tuning on CIFAR-100, all robust training settings, including both STAR modes, drop to 1.00% accuracy, which is the random guess level. Furthermore, this row is directly excluded from subsequent calculations of average robustness. What is the specific mechanism causing Adapter to collapse completely during robust training on CIFAR-100? When claiming that the method applies generally across adaptation methods, what is the rationale for excluding this failure case from the aggregate benchmark averages?
2 comments
Response to gUSX: Weaknesses
We answer the weaknesses below.
R3.W1: Novelty of the losses and of the two modes beyond two sets of loss weights.
Ans.R3.W1: The novelty of STAR lies not in its ingredients but in a finding about PEFT and the objective it leads to. (1) Under PEFT, the AT loss , i.e., the adversarial CE, stays near chance level during training, whereas STAR places its constraints on the class token, the ViT-specific summary that the classification head reads, and the same few parameters meet them: its alignment loss falls close to zero, and the newly added App. D.3 proves that its smoothness loss relaxes the AT constraint. (2) A known combination does not reproduce this: TRADES, which adds a smoothness term to the task loss, reaches 51.63% clean and 50.18% transfer robust accuracy under LoRA on CIFAR-10, against 75.50% and 71.75% for STAR-RE, and tuned AT baselines do not close the gap either, since STAR-RE trained with the same weaker attack as PGDAT stays at or above it in clean and transfer robust accuracy. (3) As in our answer to R3.Q1, STAR-RE and STAR-GE are two fixed weight settings of one objective, selected once and never re-tuned, and we do not claim them as algorithmic novelty. We add these results in Section 4.5 and App. D.3 and E.9.
R3.W2: Gap between PGD and AutoAttack for STAR-GE and necessity of each loss.
Ans.R3.W2: (1) The loss-term ablation, which removes each term in turn, shows that each loss is needed: every partial objective either fails to learn the task or learns it without robustness, and only the full objective avoids both. Adding raises the transfer robust accuracy by 8.6% even without on CIFAR-10 and from 54.33% to 59.93% under PFT on Oxford-IIIT Pet, and raises the robustness to patch attacks. (2) The PGD-AutoAttack gap of STAR-GE reflects gradient masking, which also appears for MixAT, the AT baseline at the same clean accuracy, so we now report AutoAttack on all three datasets and base every statement on it. STAR-GE targets transfer and patch robustness rather than robustness, and under LoRA on CIFAR-10 it exceeds MixAT by 3.5% under PatchFool and 2.5% under AttentionFool, two attacks on the patch and attention structure of ViTs. The -robust weight setting is STAR-RE, which under AutoAttack keeps most of the robustness of PGDAT, 19.5% against 23.8%, at 75.50% against 48.33% clean accuracy, and is over 25% more robust to both patch attacks. We add these results in Sections 4.5 and 4.7 and App. E.5.
We thank the reviewer again and hope the additions merit a higher score.
Response to gUSX: Questions
Thank you for your constructive comments. We answer the questions below.
R3.Q1: General rule for the four loss weights and a single stable setting.
Ans.R3.Q1: STAR-RE and STAR-GE are two fixed weight settings of one objective, called modes because each serves one deployment requirement; each is set once and never re-tuned. (1) Our sweep over the four weights gives the rule: λali is the largest, since alignment is the constraint that a small learnable subspace can meet, and λcle/λadv sets the trade-off, above one to keep the clean accuracy of standard adaptation against transfer attacks (STAR-GE), and at most one to gain white-box ℓ∞ robustness at a cost in clean accuracy (STAR-RE). (2) That cost is not specific to STAR: under PT and PFT on CIFAR-10, PGDAT drops further, to 24.18% and 27.63% clean accuracy against 37.56% and 47.88% for STAR-RE, which is also more robust, and over all 18 combinations of dataset and adaptation method, STAR-RE averages 70.94% clean accuracy against 25.38% for PGDAT (Table R3.1). (3) Without prior knowledge of the attack, we recommend STAR-GE as the single stable setting: unchanged on all 18 combinations, it matches the average clean accuracy of MixAT, 86.90% against 86.58%, with 4.6% and 5.1% higher transfer robust accuracy for attacks crafted on PiT-B and DeiT-B, all seven settings with λcle>λadv keep its clean accuracy within 0.24%, and its gain holds over three seeds. We state this rule in App. C.4, with the sweep in App. E.5 and the seeds in Section 4.6.
R3.Q2: Mechanism of the Adpt collapse on CIFAR-100 and its exclusion from the averages.
Ans.R3.Q2: The collapse comes from adversarial training with Adpt on CIFAR-100, not from STAR. (1) Every objective that uses adversarial examples, PGDAT, MixAT, STAR-RE and STAR-GE, collapses to the same uniform prediction, i.e., 1% chance-level accuracy, whereas the same Adpt model learns the task under standard adaptation, so the adapter itself is not the cause. We have not yet identified the exact mechanism, but we ruled out a bad starting point: after a clean warm-up, PGDAT reaches 46% validation accuracy and falls back to chance level as soon as adversarial training starts. STAR thus applies unchanged to this cell, and it is adversarial training with Adpt that fails there, for every objective alike. (2) We exclude the cell from the averages because the averages compare the objectives, and this cell gives all four robust objectives identical accuracies, so including it scales every gap by 17/18 and leaves every ranking unchanged: STAR-GE still matches MixAT in clean accuracy and is 4.6% and 5.1% more robust to transfer attacks crafted on PiT-B and DeiT-B, against 4.8% and 5.5% without it (Table R3.1). We report the cell and this analysis in App. E.1.
Table R3.1: Average accuracy (%) over 18 or 17 (w/o Adpt on CIFAR-100) combinations; Transfer: PiT-B / DeiT-B.