- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The work provides an analysis and insights on "massive activations", values within a feature map inside neural networks that are much larger than all other values. While their existence had been noted before, the authors show that 1) massive activations displace representation away from non-linearity boundaries, making them more locally linear, 2) that they co-occurr with adversarial robustness, and 3) that they give LLMs robustness to token embedding perturbations. Overall, the paper provides interesting and well-explained insights, but some statements such as "we show that massive activations are more widespread in deep learning than previously known (they do not appear only in LLMs, as was originally found in Sun et al. (2024))." are overclaimed as the original Sun paper also found massive activations in ViTs. Moreover, the relationship between massive activations and (natural) robustness was previously shown in Crabbé et al. (Interpreting CLIP: Insights on the Robustness to ImageNet Distribution Shifts)
Critical strengths
- Showing that massive activations co-ocurr with adversarial robustness (malicious examples crafted with gradient ascent), not just natural robustness (atypical examples found in real-world datasets).
- Showing that massive activations lead to robustness to embedding perturbation in LLMs
Critical weaknesses
- One of the main claims of the paper, that they show massive activations in models other than LLMs, is not true. This was already shown in Sun et al which the authors cite.
- The authors evaluate robustness with PGD only, which leaves the question of wether the robustness is real or whether it is caused by gradient masking. See Carlini et al., https://arxiv.org/abs/1902.06705, On Evaluating Adversarial Robustness.
- The authors do not cite the work of Crabbé et al. (Interpreting CLIP: Insights on the Robustness to ImageNet Distribution Shifts), which showed the relationship between massive activations and (natural) robustness.
Questions for authors
- Could the authors update their contribution statement to reflect that Sun et al. already showed massive activations in other types of models besides LLMs?
- Could the authors cite the work of Crabbé et al.?
- Is the robustness of the models shown maintained against AutoAttack? (Which contains some black-box attacks and other techniques to show gradient masking)
3 comments
Dear Reviewer 111R, we thank you once again for taking the time to review our paper.
We have invested significant time in the rebuttal to validate the adversarial robustness of the models under a stronger AutoAttack evaluation, as well as to acknowledge the related work by Crabbé et al., while highlighting our distinctive contributions, narrowing down the claims about discovering massive activations in ViTs, and have revised our manuscript to reflect these changes.
We would appreciate your feedback and that you consider raising the score.
Best regards,
The authors
Response to rebuttal
I greatly thank the authors for investing time and effort into responding to my concerns: the contributions adjustment, AutoAttack evaluation, and Crabbé et al citation successfully addressed my concerns. I will raise my score to 3.
We thank the reviewer for the thoughtful and constructive feedback. We appreciate that the reviewer found our work interesting and our insights well-explained. We also appreciate their comment on novelty in connecting massive activations with adversarial robustness rather than only natural robustness.
Additionally, your comment about adversarial evaluation with AutoAttack led to a significant improvement of the manuscript. We hope the revisions address your concerns. If any issues remain unclear or you feel further clarification is needed, we would be very happy to elaborate.
W1 and Q1: related work
We thank the reviewer for the pointer. To reflect this, we revised the contribution list in the manuscript and narrowed the wording of the abstract at lines 20-22.
W3 and Q2: related work
We thank the reviewer for pointing us to this related work, which we were unaware of. We added a dedicated discussion in our manuscript to acknowledge this related work in lines 134-142, given its relevance to our work. That said, we believe that this does not undermine the novelty and the contribution of our main claim, as we show a connection between massive activations and adversarial robustness, which involves carefully crafted perturbations to the inputs that try to cause misclassification, which we believe meaningfully differs from natural robustness as defined in Crabbé et al. Additionally, in contrast to Crabbé et al., we provide additional support for this connection across a diverse set of architectures and datasets, which includes the pretrained Pythia LLM family.
W2 and Q3: Stronger attack suite, and disentangling robustness and gradient masking
We thank the reviewer for the comment and suggestion to use AutoAttack as a way to ensure the adversarial gains are real.
To address this, we evaluated the adversarial accuracy with AutoAttack (Croce & Hein, 2023), which is constructed by the union of the APGD-CE, APGD-T, FAB-T, and Square attacks. We use 10 steps for gradient-based attacks, 2,000 Square queries, and 5 restarts for the AutoAttack configuration. To establish robustness across all the architectures in the manuscript, we re-evaluate MLP on MNIST, ResNet18 on CIFAR10, ViT Tiny on CIFAR10, ViT Small on Imagenette, GPT on Shakespeare Text, and Pythia 160M on The Pile with AutoAttack.
Throughout these different architectures and datasets, we observe that our general claim holds under the AutoAttack evaluation, and we see that massive activations and the increase in adversarial accuracy co-emerge in all architectures, in line with our main claim. We present an updated evolution of the metrics across training in updated Figure 2 (for MLP, GPT and Pythia 160M) and Figure 9 (for ResNet18, ViT Tiny and ViT Small). Importantly, we observe that for MLP, GPT, and Pythia 160M the adversarial accuracy improves substantially as depicted in the figures, while for ResNet18 and ViT architectures, the improvement is visible for smaller radii (ε=0.0025) and less dramatic (improving by 6-8%). However, the trend is present in all the cases we evaluate.
We thank the reviewer for this suggestion, as we believe that this evaluation will be a significantly valuable addition to the manuscript and will clarify to what extent the robustness of the different architectures was present under substantially stronger attacks. We will update the manuscript to reflect these new findings, and we hope that these results address your comments.
Reproducibility details
We provide the anonymous code used for this paper in the updated abstract of the manuscript for examination of the codebase.
We thank you once again for your comments and for carefully reviewing our paper. We hope that our replies and improvements to the paper have addressed your concerns, and are very happy to reply to any further points!
Given that we have invested significant effort in addressing each of your concerns through extensive new experiments, while maintaining the core strengths you recognized, we would greatly appreciate your consideration of raising the score.
References
Croce, Francesco, and Matthias Hein. "Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks." Proceedings of the 37th International Conference on Machine Learning, 2020.
Crabbé, Jonathan, et al. "Interpreting clip: Insights on the robustness to imagenet distribution shifts." Transactions on Machine Learning Research, 2023.