- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper aims to increase adversarial robustness of LoRA by proposing Attribution Sensitivity, which analyzes global attribution sharing for layers in pre-trained models and assign different LoRA ranks correspondingly to the layers with lower sensitivity instead of uniformly. Both theoretical motivation and empirical experiments are presented. The authors show that the method can increase adversarial robustness with fixed fine-tuning budgets across tested configurations.
Critical strengths
- The paper is well-written with clear section structures and plots. It is nice to read through when the motivation and method development process is clear and the results are well-organized with detailed hypotheses and analyses.
- The work is grounded with thorough background and related work across adversarial peft, parameter selection, and training allocation. As mentioned below, including designed experiments to systematically compare the proposed method with those work would make the paper stronger.
Critical weaknesses
- A core component of the proposed method is the global attribution scoring. The one mechanism adopted here is in equation 2, and its effectiveness has only been compared with uniform allocation in figure 2. More isolated and direct comparison to other scoring mechanisms discussed in related work can be interesting and helpful to see how effective the chosen one is. And it can help identify if distributed allocation is generally effective or if only a certain scoring mechanism is the key here. Also, it would be interesting to see adaptive allocation assignment instead of a fixed one since the sample’s interference effect may change as the model adapts.
- There have been studies showing the trade-off between adversarial robustness and clean accuracy in both training from scratch and fine-tuning. There is no clean classification accuracy with the proposed method reported and discussed in the paper.
- As mentioned in 3.2, the sample used in the training can lead to interference during optimization. The paper mainly tested on cifar10 and imagenet-r as downstream datasets, and as reported in table 2, the overall mean of is very low. It would be interesting to see if more diversity and specialized datasets may show different results.
Questions for authors
Thanks for submitting to ICLR 2027! The paper studies a very specific and important adversarial robustness problem in fine-tuning with strategic allocation. Addressing the following questions (based on the comments in the previous section) would help strengthen the claims.
- Many scoring mechanisms are mentioned and discussed in the paper, but I didn’t find a scoped experiments to examine how they compare with the adopted one. Do other scoring functions also increase adversarial robustness but the applied one is the most effective one? Do they make a big difference/variance?
- Can the proposed allocation be adaptive? For example, it could be focusing on the early layers as the fine-tuning starts and shifting more towards the later layers as the training progresses.
- What is the clean classification accuracy of the proposed method? What is the end-to-end overhead of the proposed method with the scoring computation and memory?