acceptodds
Withdrawn from ICLR 2027

Attribution Sensitivity Guided Parameter Allocation for Adversarially Robust Parameter-Efficient Fine-Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pre-trained foundation models that undergo standard pre-training can be efficiently fine-tuned for downstream tasks using parameter-efficient fine-tuning (PEFT) methods. However, these models remain highly vulnerable to adversarial perturbations. Although existing works provide solutions for adversarially robust PEFT, they often distribute PEFT parameters uniformly across layers, and less focus has been given to the distribution of the limited capacity of PEFT methods across model layers tailored for adversarial robustness. In this work, we introduce a novel method that guides the allocation of LoRA ranks and parameters across transformer blocks, providing better adversarial robustness compared to uniform allocation. Specifically, we re-examine a representation-level diagnostic, attribution sharing, for robust PEFT, which reveals mixed alignment between the gradient of the adversarial loss and attribution sharing. We then provide a theoretical analysis of why this mixed alignment can impede adversarial training optimization. This motivates our method, Attribution Sensitivity, a one-shot method that assigns more LoRA capacity to blocks where global attribution sharing is less sensitive to relative weight changes. Experiments on CIFAR-10 and ImageNet-R with standard-pretrained ViT-B/16 and DeiT-Tiny backbones across three rank budgets achieve the highest average robust accuracy, supporting attribution sensitivity as a useful signal for robustness-oriented LoRA rank allocation.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Adversarial Robustness, Parameter Efficient Fine Tuning, Low Rank Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hf2GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper aims to increase adversarial robustness of LoRA by proposing Attribution Sensitivity, which analyzes global attribution sharing for layers in pre-trained models and assign different LoRA ranks correspondingly to the layers with lower sensitivity instead of uniformly. Both theoretical motivation and empirical experiments are presented. The authors show that the method can increase adversarial robustness with fixed fine-tuning budgets across tested configurations.

Critical strengths
  • The paper is well-written with clear section structures and plots. It is nice to read through when the motivation and method development process is clear and the results are well-organized with detailed hypotheses and analyses.
  • The work is grounded with thorough background and related work across adversarial peft, parameter selection, and training allocation. As mentioned below, including designed experiments to systematically compare the proposed method with those work would make the paper stronger.
Critical weaknesses
  • A core component of the proposed method is the global attribution scoring. The one mechanism adopted here is in equation 2, and its effectiveness has only been compared with uniform allocation in figure 2. More isolated and direct comparison to other scoring mechanisms discussed in related work can be interesting and helpful to see how effective the chosen one is. And it can help identify if distributed allocation is generally effective or if only a certain scoring mechanism is the key here. Also, it would be interesting to see adaptive allocation assignment instead of a fixed one since the sample’s interference effect may change as the model adapts.
  • There have been studies showing the trade-off between adversarial robustness and clean accuracy in both training from scratch and fine-tuning. There is no clean classification accuracy with the proposed method reported and discussed in the paper.
  • As mentioned in 3.2, the sample used in the training can lead to interference during optimization. The paper mainly tested on cifar10 and imagenet-r as downstream datasets, and as reported in table 2, the overall mean of cos(−gi,GS)cos(-g_i, G_S) is very low. It would be interesting to see if more diversity and specialized datasets may show different results.
Questions for authors

Thanks for submitting to ICLR 2027! The paper studies a very specific and important adversarial robustness problem in fine-tuning with strategic allocation. Addressing the following questions (based on the comments in the previous section) would help strengthen the claims.

  • Many scoring mechanisms are mentioned and discussed in the paper, but I didn’t find a scoped experiments to examine how they compare with the adopted one. Do other scoring functions also increase adversarial robustness but the applied one is the most effective one? Do they make a big difference/variance?
  • Can the proposed allocation be adaptive? For example, it could be focusing on the early layers as the fine-tuning starts and shifting more towards the later layers as the training progresses.
  • What is the clean classification accuracy of the proposed method? What is the end-to-end overhead of the proposed method with the scoring computation and memory?
Reviewer_7akNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a method for LoRA rank selection in parameter-efficient fine-tuning (PEFT) to improve adversarial robustness. The Attribution Sensitivity score is motivated by mixed alignments between adversarial loss gradients and global attribution sharing gradients. Global attribution sharing measures how similarly different classes use features for prediction. The paper provides formal derivations and evaluates its method against 9 other methods on two models (ViT-B/16, DeiT-Tiny) and two datasets (Cifar10 and ImageNet-R) with PGD/APGD L_\inf=3/255.

Critical strengths
  • The proposed allocation method is evaluated against a wide range of existing rank-allocation and layer-selection strategies under matched LoRA parameter budgets.
  • The use of a fixed LoRA rank budget is a positive aspect of the problem formulation.
Critical weaknesses

1. Problem formulation and clarity

L046: “… their allocation strategies were not developed specifically for adversarial robustness.”

You introduce allocation strategies here for the first time without defining what they refer to. The next question in L047 on “LoRA rank allocation” does not clarify it either. If this is central to your work, the setup should be clear.

L047: “…given a fixed budget, can the LoRA rank allocation among transformer layers be tailored for improved adversarial robustness?”

Make it clear what kind of budget you are talking about. It only becomes somewhat clear in L080 that these are “rank budgets,” but even at that point, it is not clear what a rank budget is. The reader has to wait until the problem setup in Section 3 to get a clear definition. If this is the first work to consider such a constraint, make that clear and justify the choice, or cite works that directly define such a rank budget.

L054: “to guide LoRA capacity allocation across transformer blocks that prioritize less sharing-sensitive regions.”

Based on my previous comments on clarity, your proposition is absolutely unclear in practical terms, especially “allocation” and “less sharing-sensitive.” I can only guess what you want to do.

L064: “allocates a fixed LoRA rank”

The LoRA rank for a given layer is not variable throughout training, so specifying “fixed” here is confusing. The main point here seems to be that the allocated rank is inversely proportional to the corresponding score.

L107: “one-shot allocation of a fixed LoRA budget”

Following a previous comment on clarity, this cannot be self-defined. Does “one-shot” mean there are cases where it is iteratively chosen? By “fixed,” do you mean a single uniform value across layers?

L144: “…keeping the total LoRA parameters and total rank budget R fixed.”

In this statement, make it clear that you are talking about hyperparameters (ranks) and not the trainable LoRA A and B parameters.

2. Motivation and novelty

L051: “examine a representation-level diagnostic, attribution sharing”

What is the motivation for examining attribution sharing? There is nothing a priori connecting it to the research question you asked in the previous paragraph.

3. Definitions and interpretations

L151: “An attribution vector describes which feature coordinates contribute to the logit of that class”

This statement is an interpretation of your formula and is not true by itself. Either you keep it as a potential intuitive interpretation, or you make it a verifiable claim that you explicitly prove.

4. Related work

L085: You cannot have a related work section on robustness foundations and PEFT without mentioning Hua et al. [1], which is so far the SOTA.

L101: The section on non-adversarial scoring and allocation needs more organization. It mixes initialization, pruning, and explicit performance improvement methods that use LoRA rank selection without organizing them around rank selection and their corresponding objectives.

Moreover, you already described their objectives, so saying that they do not target downstream robustness is unnecessary, especially since it is not necessarily a problem that these methods do not target adversarial robustness. It is simply your goal to prove that your targeted method performs better.

5. Experimental evaluation and supporting evidence

L065–L070

Please add references to the datasets and models/architectures considered.

What do you mean by “standard” in “standard pretrained backbones”? If you are suggesting that this provides good coverage of architectures sufficient to support your claims, then it is wrong [1, 2]. For a better classification of architectures, check Heuillet et al. [3].

If you are already comparing across datasets and backbones (L068), why announce the two datasets and two backbones? Is there any known previous specificity?

Figure 1:

(a) It is unclear why you plot the loss and accuracy deterioration instead of the original values.

(b) Plot the values side by side instead of the relative drop. A difference between rank 0 and rank 8 does not support the claim the paper makes. I would suggest a gradual variation of ranks.

Very weak evaluation: Two datasets and two models are not sufficient to support claims about downstream adversarial robustness with robust fine-tuning.

The adversarial evaluation does not use the standard CIFAR-10 setting in the literature [4], e.g., ε=8/255 (or 4/255), making it difficult to compare with existing work.

Critical: Beyond the modest improvements in many cases in Table 1, what is the interest of allocation selection in general if the results in the whole table are nowhere near the SOTA you can get from just adversarial initialization [1]? It does not seem to be the right direction to push.

Questions for authors

L056–L057: “per-example adversarial loss gradients”

What do you mean by “per-example adversarial loss gradients” in the plural? Are you talking about vector elements (partial derivatives with respect to parameters) of an instance across optimization steps?

L152–L153: “measures the alignment of their evidence”

What do you mean by “evidence” here?

L153–L154: Global attribution sharing

You define global attribution sharing for a collection (P_0) of pairs without explicitly saying it is the set of all class pairs ((c,j)). How can it still be global if there can be so many subsets of pairs? Which one do you call “global”?

Figure 1(c):

What is “Gradient Cosine Similarity”? Have you checked that the near-zero cosine is not an effect of high dimensionality? Recall that the expected cosine between two random vectors in high dimensions is zero, with a standard deviation inversely proportional to the square root of the dimension.

L166–L167: “Importantly, this decrease in sharing is not accompanied by a corresponding deterioration in adversarial robustness.”

Where is the empirical evidence supporting this statement?

References

[1] Hua et al. (2024). Initialization Matters for Adversarial Transfer Learning. CVPR 2024. https://arxiv.org/abs/2312.05716

[2] Ngnawé et al. (2026). Robust Fine-Tuning from Non-Robust Pretrained Models: Mitigating Suboptimal Transfer With Epsilon-Scheduling. ICLR 2026. https://arxiv.org/abs/2509.23325

[3] Heuillet et al. (2025). A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy. https://arxiv.org/abs/2508.14079

[4] Croce et al. (2021). RobustBench: a standardized adversarial robustness benchmark. NeurIPS 2021. https://arxiv.org/abs/2010.09670

Reviewer_GUHNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Attribution Sensitivity, a one-shot method for allocating LoRA ranks across Transformer blocks to improve adversarial robustness. The method assigns more ranks to blocks where attribution sharing is less sensitive to weight changes. Experiments on CIFAR-10 and ImageNet-R with ViT-B/16 and DeiT-Tiny show improved robust accuracy over existing rank allocation methods under fixed parameter budgets.

Critical strengths

The paper addresses an interesting and practical problem of LoRA rank allocation for adversarial robustness. The proposed method is simple and efficient, requiring only one-shot rank allocation before adversarial training.

Critical weaknesses

Theoretical motivation: The authors propose allocating more LoRA capacity to layers with lower attribution sensitivity. From Section 3, I understand that lower attribution sensitivity leads to a smaller upper bound on attribution-sharing changes. However, the connection between lower attribution sensitivity and better adversarial robustness is still unclear to me.

Missing ablation: In addition to uniform allocation, I would like to see a comparison with direct sensitivity allocation, where more LoRA capacity is assigned to layers with higher attribution sensitivity. This would help justify the proposed inverse allocation strategy.

Early-layer allocation: In Figure 3, the authors observe that their method consistently favors earlier Transformer blocks. It would be useful to include a simple ablation comparing allocations that favor earlier versus later layers, to better understand whether the improvements come from attribution sensitivity or simply from allocating more capacity to earlier layers.

Robustness evaluation: Finally, I suggest reporting results under the full AutoAttack benchmark to provide a more comprehensive evaluation of adversarial robustness.

Questions for authors

See above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.