acceptodds
Under review as a conference paper at ICLR 2027

STAR: Multi-Level Robustness Injection for Vision Transformer Downstream Adaptation

Abstract

Pretrained Vision Transformers (ViTs) are increasingly adapted downstream by parameter-efficient fine-tuning (PEFT) under a small learnable-parameter budget . Adapted models face transfer attacks from similar ViTs, and even white-box attacks if the learnable module leaks, so robust adaptation under such a is a practical need. Across PEFT methods and full fine-tuning, we find that standard adaptation is not robust and that adversarial training (AT) cannot balance clean accuracy and robustness once shrinks. We propose Smoothness and Token Alignment Regularization (STAR), a robust adaptation algorithm that spreads the robustness signal over multiple relaxed constraints that a small learnable subspace can satisfy: label-free class-token alignment, boundary smoothness, and down-weighted cross-entropies, at the representation, distribution, and decision levels, respectively. It applies unchanged to every adaptation method. Results from multiple datasets show that the advantage of STAR over AT is more pronounced under PEFT than under full fine-tuning. Analyses also show that robustness depends more on how learnable parameters reach the Encoder than on their number.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Adversarial Robustness, Vision Transformer, Parameter-Efficient Fine-Tuning, Downstream Adaptation, Image Classification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gUSXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates adversarial robustness in downstream adaptation of pretrained Vision Transformers, termed ViT, using parameter-efficient fine-tuning, termed PEFT. The authors state that standard adaptation lacks robustness against adversarial examples. They also note that standard adversarial training, termed AT, cannot balance clean accuracy and robustness under a limited parameter budget. To address this issue, the authors propose Smoothness and Token Alignment Regularization, termed STAR. The method decomposes the adversarial objective into three components. These components include a class-token cosine alignment loss Lali\mathcal{L}_{ali} at the representation level, an interpolated token smoothness loss Lsmo\mathcal{L}_{smo} at the distribution level, and weighted cross-entropy losses at the decision level. The authors introduce two hyperparameter configurations named STAR-RE and STAR-GE. They claim that this approach achieves a better trade-off between clean accuracy and robustness under constrained parameter budgets.

Critical strengths
  1. Comprehensive experimental evaluation across diverse settings: The paper provides an extensive empirical evaluation. The authors test six downstream adaptation methods, including LP, PT, LoRA, PFT, Adapter, and FFT, alongside five training conditions across three datasets. The evaluations cover six ViT-specific black-box transfer attacks as well as several white-box attacks. In addition, the appendix includes adaptation experiments on a self-supervised DINOv2 backbone.

  2. Clear practical relevance of the problem setting: The paper addresses a common deployment scenario where practitioners adapt a shared pretrained backbone using parameter-efficient fine-tuning. It explores the security impact of restricted parameter updates. This investigation clarifies the need for robust adaptation under fixed parameter budget constraints.

Critical weaknesses
  1. Limited theoretical novelty and packaging of regular hyperparameters as distinct modes: The components of the proposed loss function are combinations of existing techniques. These include cosine distance alignment in feature space, KL divergence consistency on interpolated features, and standard cross-entropy. The method does not incorporate specialized architectural designs for the attention mechanism or patch structure of ViTs. Furthermore, the two modes highlighted in the main text, named STAR-RE and STAR-GE, are merely two manually selected sets of loss weights detailed in Appendix C.4. This reflects standard hyperparameter tuning rather than algorithmic innovation.

  2. Empirical results do not adequately demonstrate the necessity or efficacy of the proposed losses: The empirical data indicates that the proposed regularizations do not resolve the trade-off between clean accuracy and robustness. Under STAR-RE, clean accuracy drops substantially on CIFAR-10, falling from 98.37% to 37.56% under PT, and from 98.80% to 47.88% under PFT. Under STAR-GE, a large discrepancy appears between white-box PGD and standard AutoAttack evaluations. Multiple PEFT configurations drop to 0.00% robust accuracy under AutoAttack, indicating gradient masking. Overall, the experiments do not sufficiently prove that the proposed losses are necessary to establish genuine robustness.

Questions for authors
  1. Selection mechanism of hyperparameter modes and severe drops in clean accuracy: The main text describes STAR-RE and STAR-GE as two adaptation modes with different emphases. However, Appendix C.4 reveals that both modes essentially correspond to two fixed sets of loss weights. Based on Table 1, STAR-RE causes substantial drops in clean accuracy under certain PEFT configurations. For example, on CIFAR-10, clean accuracy drops by over 60 percentage points under PT, and by over 50 percentage points under PFT. Meanwhile, STAR-GE provides relatively limited robustness gains under certain evaluated attacks. In real-world downstream tasks without prior knowledge of attack types, what general rule should practitioners follow to determine the specific weights of these four loss terms? Is there a single, stable weighting configuration that achieves consistent robustness gains without severely degrading standard classification accuracy?

  2. Underlying causes of performance collapse under specific fine-tuning settings and justification for data exclusion: The paper argues that the proposed method is independent of specific parameter adaptation mechanisms and applies generally to diverse adaptation structures from LP to FFT. However, Appendix E.1 and Table 6 show that when using Adapter fine-tuning on CIFAR-100, all robust training settings, including both STAR modes, drop to 1.00% accuracy, which is the random guess level. Furthermore, this row is directly excluded from subsequent calculations of average robustness. What is the specific mechanism causing Adapter to collapse completely during robust training on CIFAR-100? When claiming that the method applies generally across adaptation methods, what is the rationale for excluding this failure case from the aggregate benchmark averages?

2 comments
AuthorsOpenReview ↗

Response to gUSX: Questions

Thank you for your constructive comments. We answer the questions below.

R3.Q1: General rule for the four loss weights and a single stable setting.

Ans.R3.Q1: STAR-RE and STAR-GE are two fixed weight settings of one objective, called modes because each serves one deployment requirement; each is set once and never re-tuned. (1) Our sweep over the four weights gives the rule: λali\lambda_{\text{ali}} is the largest, since alignment is the constraint that a small learnable subspace can meet, and λcle/λadv\lambda_{\text{cle}}/\lambda_{\text{adv}} sets the trade-off, above one to keep the clean accuracy of standard adaptation against transfer attacks (STAR-GE), and at most one to gain white-box ℓ∞\ell_\infty robustness at a cost in clean accuracy (STAR-RE). (2) That cost is not specific to STAR: under PT and PFT on CIFAR-10, PGDAT drops further, to 24.18% and 27.63% clean accuracy against 37.56% and 47.88% for STAR-RE, which is also more robust, and over all 18 combinations of dataset and adaptation method, STAR-RE averages 70.94% clean accuracy against 25.38% for PGDAT (Table R3.1). (3) Without prior knowledge of the attack, we recommend STAR-GE as the single stable setting: unchanged on all 18 combinations, it matches the average clean accuracy of MixAT, 86.90% against 86.58%, with 4.6% and 5.1% higher transfer robust accuracy for attacks crafted on PiT-B and DeiT-B, all seven settings with λcle>λadv\lambda_{\text{cle}} > \lambda_{\text{adv}} keep its clean accuracy within 0.24%, and its gain holds over three seeds. We state this rule in App. C.4, with the sweep in App. E.5 and the seeds in Section 4.6.

R3.Q2: Mechanism of the Adpt collapse on CIFAR-100 and its exclusion from the averages.

Ans.R3.Q2: The collapse comes from adversarial training with Adpt on CIFAR-100, not from STAR. (1) Every objective that uses adversarial examples, PGDAT, MixAT, STAR-RE and STAR-GE, collapses to the same uniform prediction, i.e., 1% chance-level accuracy, whereas the same Adpt model learns the task under standard adaptation, so the adapter itself is not the cause. We have not yet identified the exact mechanism, but we ruled out a bad starting point: after a clean warm-up, PGDAT reaches 46% validation accuracy and falls back to chance level as soon as adversarial training starts. STAR thus applies unchanged to this cell, and it is adversarial training with Adpt that fails there, for every objective alike. (2) We exclude the cell from the averages because the averages compare the objectives, and this cell gives all four robust objectives identical accuracies, so including it scales every gap by 17/18 and leaves every ranking unchanged: STAR-GE still matches MixAT in clean accuracy and is 4.6% and 5.1% more robust to transfer attacks crafted on PiT-B and DeiT-B, against 4.8% and 5.5% without it (Table R3.1). We report the cell and this analysis in App. E.1.

Table R3.1: Average accuracy (%) over 18 or 17 (w/o Adpt on CIFAR-100) combinations; Transfer: PiT-B / DeiT-B.

ObjectiveClean (18)Transfer (18)Clean (17)Transfer (17)
PGDAT25.3823.32 / 23.0226.8224.61 / 24.29
MixAT86.5830.80 / 22.3791.6132.53 / 23.60
STAR-RE70.9439.21 / 33.2875.0641.43 / 35.15
STAR-GE86.9035.36 / 27.5191.9537.36 / 29.05
AuthorsOpenReview ↗

Response to gUSX: Weaknesses

We answer the weaknesses below.

R3.W1: Novelty of the losses and of the two modes beyond two sets of loss weights.

Ans.R3.W1: The novelty of STAR lies not in its ingredients but in a finding about PEFT and the objective it leads to. (1) Under PEFT, the AT loss LadvL_{\text{adv}}, i.e., the adversarial CE, stays near chance level during training, whereas STAR places its constraints on the class token, the ViT-specific summary that the classification head reads, and the same few parameters meet them: its alignment loss LaliL_{\text{ali}} falls close to zero, and the newly added App. D.3 proves that its smoothness loss LsmoL_{\text{smo}} relaxes the AT constraint. (2) A known combination does not reproduce this: TRADES, which adds a smoothness term to the task loss, reaches 51.63% clean and 50.18% transfer robust accuracy under LoRA on CIFAR-10, against 75.50% and 71.75% for STAR-RE, and tuned AT baselines do not close the gap either, since STAR-RE trained with the same weaker attack as PGDAT stays at or above it in clean and transfer robust accuracy. (3) As in our answer to R3.Q1, STAR-RE and STAR-GE are two fixed weight settings of one objective, selected once and never re-tuned, and we do not claim them as algorithmic novelty. We add these results in Section 4.5 and App. D.3 and E.9.

R3.W2: Gap between PGD and AutoAttack for STAR-GE and necessity of each loss.

Ans.R3.W2: (1) The loss-term ablation, which removes each term in turn, shows that each loss is needed: every partial objective either fails to learn the task or learns it without ℓ∞\ell_\infty robustness, and only the full objective avoids both. Adding LaliL_{\text{ali}} raises the transfer robust accuracy by 8.6% even without LadvL_{\text{adv}} on CIFAR-10 and from 54.33% to 59.93% under PFT on Oxford-IIIT Pet, and LsmoL_{\text{smo}} raises the robustness to patch attacks. (2) The PGD-AutoAttack gap of STAR-GE reflects gradient masking, which also appears for MixAT, the AT baseline at the same clean accuracy, so we now report AutoAttack on all three datasets and base every ℓ∞\ell_\infty statement on it. STAR-GE targets transfer and patch robustness rather than ℓ∞\ell_\infty robustness, and under LoRA on CIFAR-10 it exceeds MixAT by 3.5% under PatchFool and 2.5% under AttentionFool, two attacks on the patch and attention structure of ViTs. The ℓ∞\ell_\infty-robust weight setting is STAR-RE, which under AutoAttack keeps most of the robustness of PGDAT, 19.5% against 23.8%, at 75.50% against 48.33% clean accuracy, and is over 25% more robust to both patch attacks. We add these results in Sections 4.5 and 4.7 and App. E.5.

We thank the reviewer again and hope the additions merit a higher score.

Reviewer_z5NTOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes STAR, a robust adaptation method for Vision Transformers under parameter-efficient fine-tuning (PEFT). The authors argue that traditional adversarial training struggles to balance clean accuracy and robustness when only a small number of parameters are trainable. To address this, STAR introduces two additional regularization terms: CLS token alignment between clean and adversarial examples, and prediction smoothness between clean and interpolated representations. Experiments across multiple datasets and PEFT methods show that STAR can achieve a better trade-off between clean accuracy and adversarial robustness.

Critical strengths
  1. The proposed method is simple and applicable to various adaptation methods.
  2. The experiments are comprehensive, covering multiple datasets, PEFT methods, and adversarial attacks.
Critical weaknesses

The main concern is that I find the motivation behind these two terms unclear. The authors argue that PEFT has a limited trainable parameter space, making the traditional adversarial CE loss difficult to optimize. However, I do not see sufficient evidence that cosine alignment is easier to optimize than adversarial CE. Similarly, for the smoothness term, it is unclear why the authors choose the midpoint of clean and adversarial representations and why optimizing its prediction should be easier. Overall, these design choices seem to be mainly based on intuition, without sufficient empirical evidence or theoretical justification.

Questions for authors

Could the authors provide stronger empirical or theoretical evidence explaining why cosine alignment and midpoint-based smoothness are easier to optimize than adversarial CE under PEFT?

1 comment
AuthorsOpenReview ↗

Response to z5NT

Thank you for the very helpful comments. We answer the questions and address the weaknesses below.

R2.Q1: Theoretical and empirical evidence that alignment and midpoint smoothness are easier to optimize than adversarial CE under PEFT.

Ans.R2.Q1: Theoretical evidence. The alignment loss LaliL_{\text{ali}} is the cosine distance between the class tokens zz and z∗z^* of a clean image and of its adversarial example, and the smoothness loss LsmoL_{\text{smo}} is the KL divergence between the predictions at their midpoint (z+z∗)/2(z + z^*)/2 and at zz. Compared with the AT loss LadvL_{\text{adv}}, i.e., the adversarial CE, two properties motivate them under restricted PEFT updates.

  1. LaliL_{\text{ali}} gives each adversarial example a concrete target that the model already handles. Whereas LadvL_{\text{adv}} only asks to raise the ground-truth class probability at the adversarial class token, LaliL_{\text{ali}} rotates the adversarial class token toward the clean class token of the same image, which the task loss LcleL_{\text{cle}} already teaches the head to classify, without needing the label or the head.
  2. LsmoL_{\text{smo}} relaxes the AT constraint by borrowing the confidence that the model already has on the clean image. Since the head is affine, the midpoint logits are the average of the clean and adversarial logits, so the midpoint stays correct even when the adversarial example is not, as long as the clean image is classified correctly by a larger logit margin than the one by which the adversarial example is misclassified. We further prove that LsmoL_{\text{smo}} never exceeds the same KL at the adversarial example, so a model that keeps the adversarial prediction close to the clean one also keeps the midpoint prediction close, but not conversely.

Empirical evidence. On CIFAR-10, we measure the loss values under all six adaptation methods and, with LoRA, ablate the position β\beta of the mixed class token βz+(1−β)z∗\beta z + (1-\beta) z^*.

  1. Under PEFT, STAR drives LaliL_{\text{ali}} close to zero, while LadvL_{\text{adv}} stays near chance level. Trained with MixAT, the model keeps LadvL_{\text{adv}} near or above chance level under every PEFT method, i.e., the loss of guessing uniformly over all classes, which is ≈2.30\approx 2.30 on CIFAR-10, and the same holds on CIFAR-100 and Oxford-IIIT Pet. STAR brings LaliL_{\text{ali}} to at most 0.092 under every PEFT method except LP, whose frozen encoder cannot align the class tokens. On the final models, STAR keeps cos⁡(z,z∗)≥0.9\cos(z, z^*) \ge 0.9 for at least 99.5% of the training examples, while no objective classifies both an image and its adversarial example correctly for more than 33%.

Table R2.1: Loss values on CIFAR-10, averaged over training.

MethodLadvL_{\text{adv}} of MixATLaliL_{\text{ali}} of STAR-RELaliL_{\text{ali}} of STAR-GE
LP3.170.930.91
PT2.730.0280.092
LoRA2.280.0050.014
PFT2.500.0250.053
Adpt2.910.0210.037
FFT1.400.0230.026
  1. Moving the point that LsmoL_{\text{smo}} constrains from the adversarial example to the midpoint improves both accuracy and robustness. From β=0\beta = 0 to β=0.5\beta = 0.5, the clean accuracy rises from 63.8% to 76.3% and the mean transfer robust accuracy from 61.52% to 71.75%, while β=0.75\beta = 0.75 adds no transfer robustness and lowers the AutoAttack robust accuracy, 23.9% against 24.3%. The midpoint thus relaxes the constraint enough to gain both accuracy and robustness, but not so far that the robustness stops improving.

Table R2.2: Accuracy (%) for each position β\beta of the mixed class token, LoRA on CIFAR-10.

β\betaCleanTransferAutoAttack
063.861.5222.7
0.2570.967.6823.8
0.5 (STAR)76.371.7524.3
0.7576.771.7523.9

We add both theoretical analysis and measurements to App. D.

We thank the reviewer again and hope the additions merit a higher score.

Reviewer_chmeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies robust adaptation of pretrained ViTs under a small learnable-parameter budget. It shows that standard adaptation and adversarial training (PGDAT, MixAT) both fail under small budgets, and proposes STAR, which replaces the hard adversarial constraint with alignment, smoothness, and reweighted cross-entropy terms. Experiments on CIFAR-10, CIFAR-100, and Oxford-IIIT Pet cover six adaptation methods and black-box and white-box attacks.

Critical strengths
  1. Treating the learnable-parameter budget ρ as a deployment constraint, and studying robustness across the full range from LP to FFT, is a useful and under-explored setting. The three research questions are stated explicitly and answered in order.

  2. The evaluation is solid. The paper uses three datasets, six adaptation methods, five training conditions, two surrogate models, multiple transfer attacks, and both PGD and AutoAttack. The use of AutoAttack to check PGD results under possible gradient masking is a good practice, and the authors state where PGD overstates robustness.

  3. The paper acknowledges that STAR-RE is not the most robust under every condition, that the Oxford-IIIT Pet gains are smaller, and that some FFT comparisons are not confirmed by AutoAttack.

  4. The DINOv2 results in Appendix E.6 test whether the conclusions hold for a self-supervised backbone, which strengthens the claim somewhat.

Critical weaknesses
  1. PGD shows signs of gradient masking, so white-box claims on CIFAR-100 and Pet are less reliable.

  2. The claim that class-token alignment drives the gain rests on one ablation (LoRA, CIFAR-10). The mechanistic arguments in Section 3.3 are heuristic.

  3. Results vary substantially with weights, and the paper gives no principled rule for choosing them

Questions for authors
  1. Can you report mean and standard deviation over at least three seeds for the main tables and ablations?

  2. How was the AT training attack strength chosen, and does tuning it change the baseline results?

2 comments
AuthorsOpenReview ↗

Response to chme: Questions

Thank you for the positive assessment. We answer the questions below.

R1.Q1: Mean and standard deviation over three seeds for main tables and ablations.

Ans.R1.Q1: We agree that variance over seeds matters for judging whether the conclusions are reliable. Given one robust training run on CIFAR takes 14 to 20 GPU-hours (App. E.4), the submission trained every model with one fixed seed (seed 1) shared by all objectives, so every comparison starts from the same initialization. Within the rebuttal period, we repeated training with three seeds (1, 2 and 3) for every objective under LoRA on CIFAR-10 and for every objective and adaptation method on Oxford-IIIT Pet. (1) Under LoRA on CIFAR-10, STAR-RE is the most robust of all five objectives against all six transfer attacks on every seed, and it beats PGDAT by at least 15.5% in clean and 14.2% in mean transfer robust accuracy, while STAR-GE beats MixAT by at least 3.2% in the latter at the same clean accuracy, (99.3 ± 0.2)% against (99.4 ± 0.2)%. (2) On Oxford-IIIT Pet, STAR-GE has the highest mean transfer robust accuracy under PT, PFT and FFT, e.g. (60.99 ± 1.51)% against (51.37 ± 2.57)% for MixAT under PFT, STAR-RE has it under LP, and STAR-GE is on par with MixAT under LoRA and Adpt, (66.78 ± 0.88)% against (66.84 ± 0.51)% and (44.26 ± 4.77)% against (44.52 ± 5.44)%. We add these results as the new Section 4.6, and we will continue to add seeds for the ablations.

R1.Q2: Choice of the AT attack strength and effect of tuning it on the baselines.

Ans.R1.Q2: We used ϵtrain=8/255\epsilon_{\mathrm{train}} = 8/255 with PGD-10 by default, the standard setting of Madry et al. (2018), for every objective with an AT loss LadvL_{\text{adv}}, so that only the objective differs. We agree that this choice should be tested, so we added an ablation with LoRA on CIFAR-10 over ϵtrain∈{8/255,4/255,2/255}\epsilon_{\mathrm{train}} \in \{8/255, 4/255, 2/255\}, in which PGDAT and STAR-RE are retrained with the same ϵtrain\epsilon_{\mathrm{train}}, and over the weight of LadvL_{\text{adv}} in MixAT, 0.25, 0.5 and 0.75. Tuning changes the baselines but not the conclusion: a weaker attack raises the clean and transfer robust accuracy of PGDAT, but STAR-RE trained with the same attack stays at or above it in both at every ϵtrain\epsilon_{\mathrm{train}}, e.g. 90.4% and 83.45% against 69.6% and 66.93% at 4/2554/255, and reweighting the LadvL_{\text{adv}} of MixAT does not close the gap either: with a weight of 0.25, it stays 5.56% less robust to transfer attacks than STAR-GE at the same clean accuracy, and with 0.75, it stays 23.3% and 20.6% below STAR-RE in clean and transfer robust accuracy. We add this ablation to Section 4.5.

AuthorsOpenReview ↗

Response to chme: Weaknesses

We answer the weakness below.

R1.W1: Gradient masking analysis of PGD on CIFAR-100 and Oxford-IIIT Pet.

Ans.R1.W1: Comparing PGD with AutoAttack shows that, on CIFAR-100 and Oxford-IIIT Pet, gradient masking appears mainly for models that keep the clean accuracy of standard adaptation, such as those of MixAT and STAR-GE, so PGD alone overstates their ℓ∞\ell_\infty robustness. To expose this, we additionally evaluated all 60 models of both datasets with standard AutoAttack at ϵ=8/255\epsilon = 8/255, which is designed to reveal gradient masking, and report it alongside PGD. On CIFAR-100, the white-box conclusion holds: under LoRA, STAR-RE is the only objective that keeps both clean accuracy far above chance level and ℓ∞\ell_\infty robustness, 9.9% at 45.0% clean accuracy, and under FFT it keeps 16.9% against 19.6% for PGDAT at 22.8% higher clean accuracy, so AutoAttack confirms most of its PGD robust accuracy. On Oxford-IIIT Pet, no objective, the AT baselines included, keeps more than 5.5% under AutoAttack with any PEFT method, consistent with robust generalization requiring more data (Schmidt et al., 2018), so we compare the objectives there by transfer robustness, which does not use the gradient of the target model and where STAR leads or matches the best baseline, as in our answer to R1.Q1. We now base every ℓ∞\ell_\infty statement on AutoAttack, as the new Section 4.7 reports.

R1.W2: Mechanistic evidence for class-token alignment LaliL_{\text{ali}} across datasets and adaptation methods.

Ans.R1.W2: (1) Beyond our original ablation with LoRA on CIFAR-10, we repeated it on Oxford-IIIT Pet under PFT, where adding the alignment loss LaliL_{\text{ali}} to the task loss LcleL_{\text{cle}} and the AT loss LadvL_{\text{adv}} raises the transfer robust accuracy from 54.33% to 59.93%. On CIFAR-10, the same holds even without LadvL_{\text{adv}}: adding LaliL_{\text{ali}} to LcleL_{\text{cle}} and the smoothness loss LsmoL_{\text{smo}} raises the transfer robust accuracy from 11.93% to 20.53% at the same clean accuracy. (2) To formalize why alignment is easier to satisfy in a small learnable subspace, the newly added App. D.3 shows that LaliL_{\text{ali}} involves neither the classification head nor the label, so its gradient never moves the decision boundaries, and proves that the smoothness constraint of LsmoL_{\text{smo}} contains and relaxes the AT constraint. The training loss values support this: without LsmoL_{\text{smo}} and LaliL_{\text{ali}}, LadvL_{\text{adv}} stays near chance level under every PEFT method on all three datasets, whereas on CIFAR-10 STAR drives LaliL_{\text{ali}} to at most 0.092 under every PEFT method that adapts the encoder, while LP serves as a control: its frozen encoder cannot align the class tokens, so LaliL_{\text{ali}} stays near 0.92 and STAR performs on par with the objective without LaliL_{\text{ali}}, showing that the robustness gain of STAR appears exactly where LaliL_{\text{ali}} can be reduced. We add these results to Section 4.5.

R1.W3: Sensitivity to the loss weights and a principled rule for choosing them.

Ans.R1.W3: (1) We agree that the weights matter, but the defender sets them once by the deployment requirement and never re-tunes them. We see two common requirements in practice: a deployed model that must keep the clean accuracy of standard adaptation and mainly faces transfer attacks crafted on public ViTs, served by STAR-GE, and a model that needs white-box ℓ∞\ell_\infty robustness and can trade clean accuracy for it, served by STAR-RE. (2) The weights express this choice by one ratio, λcle/λadv\lambda_{\text{cle}}/\lambda_{\text{adv}} above one for STAR-GE and at most one for STAR-RE, while both give LaliL_{\text{ali}} the largest weight, since alignment is the constraint that a small budget can satisfy. No further tuning is needed: we selected both settings once and used them unchanged on all 18 combinations of dataset and adaptation method and on DINOv2, where STAR-GE matches the average clean accuracy of MixAT, 86.90% against 86.58%, with about 5% higher transfer robust accuracy, while STAR-RE keeps most of the AutoAttack robustness of PGDAT at much higher clean accuracy, as in our answer to R1.W1. The exact weights matter little: all seven settings with λcle>λadv\lambda_{\text{cle}} > \lambda_{\text{adv}} keep the clean accuracy within 0.24%. For an unknown threat, we recommend STAR-GE, and we state this rule in App. C.4.

We thank the reviewer again and hope the additions merit a higher score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.