acceptodds
Under review as a conference paper at ICLR 2027

Understanding the Importance of Massive Activations Through Adversarial Robustness

Abstract

A common occurrence in Large Language Models (LLMs) is the emergence of so-called massive activations. This phenomenon is well known in the literature and has been connected to attention sinks, representational compression, as well as signal propagation. However, the reasons and consequences for their appearance are not yet well understood. In this paper, we explain the role of massive activations in deep learning by showing that they are connected to the region migration phenomenon in delayed robustness. Under the spline theoretic interpretation of deep networks, we show that massive activations push the representations away from the nonlinearity boundaries of the network, decreasing the complexity of its input-output map around data points. Furthermore, we uncover for the first time that this phenomenon is not unique to LLMs, and actually appears in a number of common architectures performing simple supervised learning tasks. We validate this connection through extensive experiments on a diverse set of architectures including residual MLP, ResNet18, Vision Transformer and GPT and datasets including MNIST, CIFAR10, Imagenette and Shakespeare Text. We consistently observe that massive activations and region migration co-occur with adversarial robustness. Furthermore, we illustrate that this connection holds in the large scale pretraining of LLMs, by showing that the raise of massive activations in LLMs closely tracks the adversarial accuracy of the model under perturbations of the token embeddings.

Primary area:
interpretability and explainable AI
Keywords:
grokking, adversarial robustness, spline theory, interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_111ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work provides an analysis and insights on "massive activations", values within a feature map inside neural networks that are much larger than all other values. While their existence had been noted before, the authors show that 1) massive activations displace representation away from non-linearity boundaries, making them more locally linear, 2) that they co-occurr with adversarial robustness, and 3) that they give LLMs robustness to token embedding perturbations. Overall, the paper provides interesting and well-explained insights, but some statements such as "we show that massive activations are more widespread in deep learning than previously known (they do not appear only in LLMs, as was originally found in Sun et al. (2024))." are overclaimed as the original Sun paper also found massive activations in ViTs. Moreover, the relationship between massive activations and (natural) robustness was previously shown in Crabbé et al. (Interpreting CLIP: Insights on the Robustness to ImageNet Distribution Shifts)

Critical strengths
  • Showing that massive activations co-ocurr with adversarial robustness (malicious examples crafted with gradient ascent), not just natural robustness (atypical examples found in real-world datasets).
  • Showing that massive activations lead to robustness to embedding perturbation in LLMs
Critical weaknesses
  • One of the main claims of the paper, that they show massive activations in models other than LLMs, is not true. This was already shown in Sun et al which the authors cite.
  • The authors evaluate robustness with PGD only, which leaves the question of wether the robustness is real or whether it is caused by gradient masking. See Carlini et al., https://arxiv.org/abs/1902.06705, On Evaluating Adversarial Robustness.
  • The authors do not cite the work of Crabbé et al. (Interpreting CLIP: Insights on the Robustness to ImageNet Distribution Shifts), which showed the relationship between massive activations and (natural) robustness.
Questions for authors
  • Could the authors update their contribution statement to reflect that Sun et al. already showed massive activations in other types of models besides LLMs?
  • Could the authors cite the work of Crabbé et al.?
  • Is the robustness of the models shown maintained against AutoAttack? (Which contains some black-box attacks and other techniques to show gradient masking)
3 comments

We thank the reviewer for the thoughtful and constructive feedback. We appreciate that the reviewer found our work interesting and our insights well-explained. We also appreciate their comment on novelty in connecting massive activations with adversarial robustness rather than only natural robustness.

Additionally, your comment about adversarial evaluation with AutoAttack led to a significant improvement of the manuscript. We hope the revisions address your concerns. If any issues remain unclear or you feel further clarification is needed, we would be very happy to elaborate.

W1 and Q1: related work

We thank the reviewer for the pointer. To reflect this, we revised the contribution list in the manuscript and narrowed the wording of the abstract at lines 20-22.

W3 and Q2: related work

We thank the reviewer for pointing us to this related work, which we were unaware of. We added a dedicated discussion in our manuscript to acknowledge this related work in lines 134-142, given its relevance to our work. That said, we believe that this does not undermine the novelty and the contribution of our main claim, as we show a connection between massive activations and adversarial robustness, which involves carefully crafted perturbations to the inputs that try to cause misclassification, which we believe meaningfully differs from natural robustness as defined in Crabbé et al. Additionally, in contrast to Crabbé et al., we provide additional support for this connection across a diverse set of architectures and datasets, which includes the pretrained Pythia LLM family.

W2 and Q3: Stronger attack suite, and disentangling robustness and gradient masking

We thank the reviewer for the comment and suggestion to use AutoAttack as a way to ensure the adversarial gains are real.

To address this, we evaluated the adversarial accuracy with AutoAttack (Croce & Hein, 2023), which is constructed by the union of the APGD-CE, APGD-T, FAB-T, and Square attacks. We use 10 steps for gradient-based attacks, 2,000 Square queries, and 5 restarts for the AutoAttack configuration. To establish robustness across all the architectures in the manuscript, we re-evaluate MLP on MNIST, ResNet18 on CIFAR10, ViT Tiny on CIFAR10, ViT Small on Imagenette, GPT on Shakespeare Text, and Pythia 160M on The Pile with AutoAttack.

Throughout these different architectures and datasets, we observe that our general claim holds under the AutoAttack evaluation, and we see that massive activations and the increase in adversarial accuracy co-emerge in all architectures, in line with our main claim. We present an updated evolution of the metrics across training in updated Figure 2 (for MLP, GPT and Pythia 160M) and Figure 9 (for ResNet18, ViT Tiny and ViT Small). Importantly, we observe that for MLP, GPT, and Pythia 160M the adversarial accuracy improves substantially as depicted in the figures, while for ResNet18 and ViT architectures, the improvement is visible for smaller radii (ε=0.0025) and less dramatic (improving by 6-8%). However, the trend is present in all the cases we evaluate.

We thank the reviewer for this suggestion, as we believe that this evaluation will be a significantly valuable addition to the manuscript and will clarify to what extent the robustness of the different architectures was present under substantially stronger attacks. We will update the manuscript to reflect these new findings, and we hope that these results address your comments.

Reproducibility details

We provide the anonymous code used for this paper in the updated abstract of the manuscript for examination of the codebase.

We thank you once again for your comments and for carefully reviewing our paper. We hope that our replies and improvements to the paper have addressed your concerns, and are very happy to reply to any further points!

Given that we have invested significant effort in addressing each of your concerns through extensive new experiments, while maintaining the core strengths you recognized, we would greatly appreciate your consideration of raising the score.

References

Croce, Francesco, and Matthias Hein. "Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks." Proceedings of the 37th International Conference on Machine Learning, 2020.

Crabbé, Jonathan, et al. "Interpreting clip: Insights on the robustness to imagenet distribution shifts." Transactions on Machine Learning Research, 2023.

Dear Reviewer 111R, we thank you once again for taking the time to review our paper.

We have invested significant time in the rebuttal to validate the adversarial robustness of the models under a stronger AutoAttack evaluation, as well as to acknowledge the related work by Crabbé et al., while highlighting our distinctive contributions, narrowing down the claims about discovering massive activations in ViTs, and have revised our manuscript to reflect these changes.

We would appreciate your feedback and that you consider raising the score.

Best regards,

The authors

Reviewer_111ROpenReview ↗

Response to rebuttal

I greatly thank the authors for investing time and effort into responding to my concerns: the contributions adjustment, AutoAttack evaluation, and Crabbé et al citation successfully addressed my concerns. I will raise my score to 3.

Reviewer_Ca2dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes that massive hidden activations drive migration away from activation boundaries and accompany increased adversarial robustness. It combines sufficient geometric conditions for residual ReLU networks, training trajectories across vision and language models, and activation-suppression interventions. The restricted argument and interventions offer useful evidence. However, the interpretation of Transformer local complexity, the relevance of the theoretical conditions, and the adequacy of the attacks remain insufficiently established to support the broad mechanistic conclusions.

Critical strengths
  • The dominance/alignment bound identifies a precise way that a representation can avoid ReLU boundaries. Its conditions are feasible, and the residual-stream argument is valid after a small indexing correction.

  • The MLP and ViT suppression experiments directly manipulate activations and inspect local geometry. Our independent MLP checks also found some LC increases under suppression, supporting the plausibility of the proposed effect while showing that selection and controls matter.

Critical weaknesses
  • The theoretical assumptions are unmeasured, while Transformer LC lacks a precise implementation and justified affine-region interpretation. The broad mechanistic conclusion therefore exceeds the restricted theorem and available intervention evidence.

  • Short single-start attacks, absent convergence/uncertainty analysis, and activation-only evidence for most LLM families cannot establish the claimed broad connection reliably. Missing code and configurations prevent checking whether stronger attacks or controlled interventions preserve the reported conclusions.

Questions for authors
  1. Which tensors and thresholds define LC for GELU, attention, and normalization? Is LC a sign-crossing proxy or an implemented spline approximation, and what validates its geometric interpretation?
  2. How often do the paired dominance/alignment assumptions and magnitude bound hold in the actual checkpoints? Please correct the initial-state indexing and identify precisely which network segment the theorem covers.
  3. Do the conclusions survive longer multi-start attacks and alternative losses under fully specified norms, preprocessing, projection, and sample counts? Please provide the LLM evaluation data and normalized interpretation of embedding perturbations.
  4. How are dominant coordinates selected? What are the LC, clean-accuracy, and adversarial-accuracy changes under dominant, random, similarly salient, and norm-matched interventions? Please supply the referenced source, configurations, and raw trajectories needed to check these comparisons.
4 comments

Reply 1/3

We thank the reviewer for the feedback and engagement with our work.

Several of your comments led directly to substantive improvements in the manuscript, particularly the comments about the adversarial evaluations with different loss functions and contextualizing the norm of LLM embeddings relative to the adversarial radius.

We now respond in detail.

On the implementation of LC in transformers

In transformer architectures, our LC implementation counts the number of MLP preactivations whose sign changes across the sampled frame. The intuition comes from ReLU networks, where a preactivation crossing zero marks a change in the neuron’s affine map. However, GELU is not a continuous piecewise affine function, hence it does not have a breakpoint, so, we use ReLU as a rough approximation of GELU and use the same sign-change count. We decided to not take into account the attention and normalization operations as their spline approximation is non-trivial to define. However, below we show that our LC proxy is adequate.

On the interpretation and validation of LC in transformers

To check what this proxy captures, we asked whether examples with higher LC also have larger errors under a local linear approximation. On the same frame used to compute LC, we used the Jacobian at the center to predict the logits at the perturbed points and measured the absolute RMS error. Presented in the table below, the Spearman correlations across 64 examples at the early, middle, and final checkpoints were 0.86, 0.66, and 0.52 for ViT Tiny, 0.85, 0.78, and 0.16 for GPT, and 0.69, 0.57, and 0.79 for Pythia 160M. The observed correlations were positive at every tested checkpoint and radius, meaning that LC is positively correlated with how locally linear the transformer is, making it an adequate proxy for measuring local nonlinearity. We hope that this addresses the reviewer’s comment. We will add this clarification in the updated version of the manuscript.

ModelTraining stepρ across five radii
ViT Tiny10.86
1,0070.66
1,000,0000.52
GPT10.85
1,1840.78
499,9990.16
Pythia 160M160.69
4,0000.57
143,0000.79

W1 and Q2: measuring the theoretical assumptions

On measuring the theoretical assumptions

To show that the trained networks satisfy the theoretical assumptions stated in Theorem 3.2, we take checkpoints from the MLP MNIST training run and measure the frequency of the massive activation dominance and weight non-cancelation assumptions hold and report the results in the table below. We observe that throughout training, the assumptions get satisfied for more and more examples, alongside the massive activations increasingly satisfy the required bound for delta=0.005 during the training (as depicted in the last column). We highlight that the theoretical setting provides pessimistic guarantees which can be improved. The current theory does not take into account that the representation can contain multiple massive activations and assumes that there is a single one. We note that the current theoretical setup can be naturally extended to multiple massive activations with similar guarantees. We will make this adjustment in the final version of the manuscript to clarify this point.

Training stepDominance (% of image–block pairs)Dominance and non-cancellation (% of image–neuron pairs)Distance > 0.005 certified (% of image–neuron pairs)
10.13%0%0%
1210.58%0%0%
11713.59%0.017%0.002%
1,08221.53%0.187%0.051%
3,85673.75%7.88%5.50%
7,27875.50%9.55%6.37%

On the subnetwork the theory covers

Regarding the question about which segment of the network the theorem covers, we clarify that the theoretical argument claims if the massive activation emerges at layer \ell then that the local geometry is simplified for the subsequent layers of the network. As the manuscript mentions, in practice the massives typically emerge in the first layer, so it is applied to layers and the subnetwork onwards from that point. We will make these clarifications in the updated version of the manuscript.

Reply 2/3

W2 and Q3: Stronger attack suite and further evaluation details

On the adversarial evaluation protocol

Firstly, we would like to clarify that for radius ε (measured in the input space, i.e. in the [0, 1] pixel space) we are using PGD attack with L_{\infty} norm, meaning the perturbation entries are clipped to be within [-ε, ε].

On a stronger attack suite

Regarding the comment about multi-start attacks on alternative losses, we evaluated adversarial accuracy with AutoAttack (Croce & Hein, 2023), a comprehensive attack suite including APGD-CE, APGD-T, FAB-T and Square attacks.

We use 10 steps for gradient-based attacks, 2,000 Square queries and 5 restarts for the AutoAttack configuration. To establish the robustness across all the architectures in the manuscript we evaluated MLP on MNIST, ResNet18 on CIFAR10, ViT Tiny on CIFAR10, ViT Small on Imagenette, GPT on Shakespeare Text and Pythia 160M on The Pile with AutoAttack. Crucially, we observe that our general claim holds under the AutoAttack evaluation, and we see that massive activations and the increase in adversarial accuracy co-emerge in all architectures. We present an updated evolution of the metrics across training in updated Figure 2 (for MLP, GPT and Pythia 160M) and Figure 9 (for ResNet18, ViT Tiny and ViT Small). Importantly, we observe that for MLP, GPT and Pythia 160M the adversarial accuracy is improving substantially as depicted in the figures, while for ResNet18 and ViT architectures, the improvement is visible for smaller radii (ε=0.0025) and less dramatic (improving by 6-8%), however the trend is still present.

Additionally, to measure the persistence of the results to multistep attacks we evaluate GPT with AutoAttack using 100 steps for gradient-based attacks. We observe that the adversarial accuracy holds up even for the 100 step AutoAttack at 56.96% vs 10 step AutoAttack at 57.24%. We hope that this addresses the reviewers comment on multistep attacks.

On LLM evaluation data and normalized interpretation of embedding perturbations

For Pythia 160M, we use the first 64 examples from the EleutherAI/the_pile_deduplicated (HuggingFace dataset) training split, truncated to 64 tokens. Perturbations are applied to the input token embeddings under an absolute L_{\infty} budget and every embedding coordinate at every token can change by at most ε. To put these budgets in context, we measured the mean per-token embedding norms on the evaluation samples at the final checkpoints. For GPT, the L_{\infty} norm is 2.738 and for Pythia 160M 0.146. GPT’s perturbation budget ε = 0.03 is therefore 1.10% of its mean embedding L_{\infty} norm, while Pythia’s budgets ε = 0.0025 1.71% of its mean embedding norm. Since an L_{\infty} budget ε also bounds each token’s perturbation L_{2} norm by ε\sqrt{d}, these budgets allow perturbation L_{2} norms of at most 4.44% of the mean embedding L_{2} norm for GPT, and 9.74% and 19.49% for Pythia. These percentages show the L_{\infty} and L_{2} norm of the perturbation compared to the average embedding norm, putting the chosen ε values into perspective as their absolute scale alone is not sufficiently informative.

We believe that this evaluation would be a significantly valuable addition to the manuscript and would clarify to what extent the robustness of the different architectures was present under substantially stronger attacks and put the LLM embedding norm into perspective. We will update the manuscript to reflect these new findings and we hope that these results addressed your comments.

Reply 3/3

W2 and Q4: selection of dominant coordinates, tracking clean accuracy, adversarial accuracy and LC under different interventions, and reproducibility details

On the selection of dominant coordinates

For MLPs we select dominant activations separately per class, since it has class specific dominant activations as discussed in the Appendix D.2. More specifically, the massive activation coordinates are identified for each class sample separately for the suppression experiments in Section 3.3 and Appendix D.3.

On tracking clean accuracy, adversarial accuracy, and LC changes

We thank the reviewer for raising this point. To address it, we track the clean accuracy, adversarial consistency (which measure the frequency the model prediction change, under adversarial attack) and LC (proxy) when we zero out the massive activations in ViT Tiny, GPT and Pythia 160M models vs randomly selected activations.

ModelConditionClean accuracy (%)Adversarial consistency (%)LC
ViT Tiny (ε = 0.0025)No suppression71.8839.8447.55
Top 4 zeroed59.3821.8861.97
Random 4 zeroed71.7238.7547.20
GPT (ε = 0.1)No suppression62.8439.4911.65
Top 4 zeroed22.944.1943.09
Random 4 zeroed63.0539.1611.62
Pythia 160M (ε = 0.0025)No suppression38.2613.212,483.28
Top 6 zeroed18.414.553,255.83
Random 6 zeroed38.1513.022,517.70

The table represents that the consistency is reduced when the massive activations are removed from the model and is unchanged when instead the same number of random activations are removed. This trend is accompanied by the corresponding increase in LC across all the models. Importantly, we also notice that the clean accuracy of the model is also dropped when massives are removed, this is expected and consistent with prior work and does not undermine or contradict the consistency result, because it measures the robustness of the model, which is the prediction change under the attack, regardless if the original prediction is correct or not. We hope that this result addresses the reviewer’s comment and establishes that this effect is specific to massive activations.

We would like to note that the norm matching experiment is not feasible to conduct, since massive activations account for the majority of the representational norm, hence another set of activations would not exist that share the representational norm of the massives.

Reproducibility details

We provide the anonymous code used for this paper in the updated abstract of the manuscript, which also includes the suppression experiment scripts for independent experiment evaluations.

We thank you once again for your valuable comments and for carefully reviewing our paper. We hope that our replies and improvements to the paper have addressed your concerns and we are very happy to reply to any further points! Given that we have invested significant effort in addressing each of your concerns through extensive new experiments, while maintaining the core strengths you recognized, we would greatly appreciate your consideration for raising the score.

References

Croce, Francesco, and Matthias Hein. "Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks." Proceedings of the 37th International Conference on Machine Learning, 2020.

Dear Reviewer Ca2d, we thank you once again for taking the time to check our manuscript.

We have invested significant time in the rebuttal to address the issues you raised regarding the interpretation and adequacy of LC metric for transformer architectures, stronger adversarial attacks, interpretation of adversarial radius in autoregressive architectures, controlled suppression study on massive activations, and its impact on LC and consistency.

Given some of the additional changes we carried out in the rebuttal, we would appreciate your feedback and that you consider raising the score.

Best regards,

The authors

Reviewer_SEweOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the role of massive activations through their relationship with region migration and adversarial robustness. For residual ReLU networks, the authors derive sufficient conditions under which dominant activations move representations away from activation boundaries. They then examine activation magnitudes, local complexity (LC), and adversarial accuracy throughout training across residual MLPs, ResNet18, ViTs, GPT, and Pythia checkpoints. Activation-suppression experiments further illustrate how dominant activations affect local partition geometry.

Critical strengths
  1. A meaningful research question examined across different network types. The paper investigates a possible functional role for massive activations rather than simply documenting their existence. Studying visual classification, character-level language modeling, and Pythia pretraining makes the proposed connection relevant beyond a single model or task.

    1. Intuitive geometric analysis supported by interventions. Proposition 3.1 and Theorem 3.2 provide explicit sufficient conditions in the residual ReLU setting. The suppression experiments in Section 3.3 and Appendix D.3 further show that modifying dominant activations changes LC and partition geometry. This goes beyond correlations between training curves and is one of the more compelling parts of the paper.
Critical weaknesses
  1. The robustness conclusions require a more thorough adversarial evaluation. Appendix C.2 primarily uses 10-step PGD with a single random start, and Section 4.1 also uses 10-step PGD for Pythia. The attack norm, the relationship between perturbation budgets and input preprocessing, and sensitivity to attack steps, step sizes, or restarts are not clearly established.
  2. The evidence does not yet sufficiently connect geometric changes to robustness, and the interpretation of LC in Transformers remains unclear.
Questions for authors

1.Attack evaluation. Please specify the attack norm, preprocessing, and projection constraints. For language models, is the budget imposed per token or over the entire embedding sequence? Do the main robustness trends persist on representative checkpoints with more PGD steps, tuned step sizes, and multiple restarts?

2.Intervention results. How do clean accuracy, adversarial accuracy, and LC change across the suppression levels in Figure 11? Could an appropriate non-dominant-activation intervention or a control matched for representational change help establish whether the effect is specific to massive activations?

3.LC implementation. Which boundaries are counted in Pythia and ViT? Is a piecewise-affine approximation used, or does the procedure count preactivation sign changes? If LC is a proxy in these models, please clarify its interpretation and provide a representative validation that it reflects changes in local nonlinearity.

4.Contribution and definition. Section 5 of Sun et al. (2024) already studies massive activations in ViTs, so the abstract’s claim of the first discovery that they are not unique to LLMs should be narrowed. What is the additional finding on this point? Also, how does tracking the absolute magnitudes of the top three activations distinguish the emergence of massive activations from an increase in overall activation scale?

3 comments

Reply 1/2

We thank the reviewer for the thoughtful and constructive feedback. We appreciate the reviewer’s recognition of the meaningful research question, the relevance of our approach, and the persuasiveness of our results. We now address the rest of the Reviewer’s comments and requests:

W1 and Q1: More thorough adversarial evaluation and details:

On the adversarial evaluation protocol

For radius ε (measured in the input space, i.e. in the [0, 1] pixel space) we are using PGD attack with L_{\infty} norm, meaning the perturbation entries are clipped to be within [-ε, ε]. For the GPT and Pythia architectures the budget is imposed on the entire embedding sequence and every perturbation coordinate at every token is clipped to be within [-ε, ε].

On multistep, tuned step size and multiple restarts attacks

Regarding the evaluation with more PGD steps, tuned step size and multiple restarts, in the table below we present the MLP, ViT Tiny and GPT architectures’ final checkpoint (when the delayed robustness state is reached) evaluated with 50 PGD steps, a few different step sizes and 5 restarts.

EvaluationMLP on MNIST (step 7,278), ε = 0.03ViT Tiny on CIFAR-10 (step 1,000,000), ε = 0.0025GPT on Shakespeare (step 499,999), ε = 0.03
Clean accuracy80.47%71.88%62.84%
PGD-10, one random start68.75%44.53%58.56%
PGD-50, five random starts, step size 0.05ε67.97%44.53%58.31%
PGD-50, five random starts, step size 0.15ε65.63%42.97%57.86%
PGD-50, five random starts, step size 0.30ε64.06%42.97%57.71%
PGD-50, union across the three step sizes64.06%42.97%57.67%

The final row represents the union of all the attacks above (the union is defined as applying all the attacks individually for each input and taking the most successful attempt on that input). We observe that attacks with more steps, restart and tuned step sizes only change the adversarial accuracy by a few percentages for the architecture, which confirms that models are robust under stronger attacks.

On more thorough adversarial evaluation

Similar to your comments Reviewers Ca2d and 111R also asked whether adversarial robustness would hold under stronger attacks. We have conducted additional adversarial evaluations using AutoAttack. Please see the detailed description and results of it under the section W2 and Q3 of our response to Reviewer 111R.

Q2: Intervention results

On tracking clean accuracy, adversarial accuracy, and LC changes

We thank the reviewer for raising this point. We find that removing some activations strongly hinders clean accuracy, making adversarial accuracy meaningless (since it measures the stability of correct predictions). Therefore, we track adversarial consistency, which measures the frequency of changes in the model’s predictions under adversarial attack. This measurement, similarly to adversarial accuracy, quantifies robustness to perturbations but does not require a meaningful starting clean accuracy.

We report the experiment requested by the reviewer, tracking clean accuracy, adversarial consistency and LC when we zero out the massive activations in ViT Tiny, GPT and Pythia 160M models vs randomly selected activations.

ModelConditionClean accuracy (%)Adversarial consistency (%)LC
ViT Tiny (ε = 0.0025)No suppression71.8839.8447.55
Top 4 zeroed59.3821.8861.97
Random 4 zeroed71.7238.7547.20
GPT (ε = 0.1)No suppression62.8439.4911.65
Top 4 zeroed22.944.1943.09
Random 4 zeroed63.0539.1611.62
Pythia 160M (ε = 0.0025)No suppression38.2613.212,483.28
Top 6 zeroed18.414.553,255.83
Random 6 zeroed38.1513.022,517.70

The table shows that adversarial consistency drops when massive activations are removed from the model, and remains unchanged when the same number of random activations are removed. This trend is accompanied by the corresponding increase in LC across all the models. These results provide further evidence for the causal and mechanistic role of massive activations in adversarial robustness, strengthening our proposal.

Reply 2/2

W2 and Q3: Connecting geometric changes to robustness, and the implementation of LC in transformers.

On the implementation of LC in transformers

In transformer architectures, our LC implementation counts the number of MLP preactivations whose sign changes across the sampled frame. The intuition comes from ReLU networks, where a preactivation crossing zero marks a change in the neuron’s affine map. However, GELU is not a continuous piecewise affine function, hence it does not have a breakpoint, so, we use ReLU as a rough approximation of GELU and use the same sign-change count.

On the interpretation and validation of LC in transformers

To check what this proxy captures, we asked whether examples with higher LC also have larger errors under a local linear approximation. On the same frame used to compute LC, we used the Jacobian at the center to predict the logits at the perturbed points and measured the absolute RMS error. Presented in the table below, the Spearman correlations across 64 examples at the early, middle, and final checkpoints were 0.86, 0.66, and 0.52 for ViT Tiny, 0.85, 0.78, and 0.16 for GPT, and 0.69, 0.57, and 0.79 for Pythia 160M. The observed correlations were positive at every tested checkpoint and radius, meaning that LC is positively correlated with how locally linear the transformer is, making it an adequate proxy for measuring local nonlinearity. We hope that this addresses the reviewer’s comment. We will add this clarification in the updated version of the manuscript.

ModelTraining stepρ across five radii
ViT Tiny10.86
1,0070.66
1,000,0000.52
GPT10.85
1,1840.78
499,9990.16
Pythia 160M160.69
4,0000.57
143,0000.79

Q4: Contribution and definition

On related work

We thank the reviewer for the pointer. To reflect this, we revised the manuscript and narrowed the wording of the abstract at lines 20-22. Importantly, our main contribution is providing the mechanistic connection of massive activations emerging during training to spline geometry and emergent adversarial robustness in deep networks, which meaningfully differs from Sun et al. that provide the existence of the massive activations and no concrete description of their function in the network.

On massive activations vs. overall activation scale

In our answer to Q2, we showed that removing specifically massive activations has a measurably different effect on adversarial robustness and LC compared to random activations. We believe that these results strengthen the causal role of massive activations vs. a general increase in activation scale, as removing random activations did not affect adversarial robustness or LC.

Reproducibility details

We provide the anonymous code used for this paper in the updated abstract of the manuscript, for examination of the codebase. We thank you once again for your comments and for carefully reviewing our paper.

We hope that our replies and improvements to the paper have addressed your concerns and we are very happy to reply to any further points! Given that we have invested significant effort in addressing each of your concerns through extensive new experiments, while maintaining the core strengths you recognized, we would greatly appreciate your consideration for raising the score.

Dear Reviewer SEwe, thank you again for reviewing and your thorough feedback.

We have invested significant time in the rebuttal to address the raised issues regarding more thorough adversarial evaluation, adequacy of LC for transformers, as well as a controlled suppression study on massive activations to validate their connection to LC and consistency.

We would appreciate your feedback and that you consider raising the score.

Best regards,

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Reviewers, ACs and PCs,

We thank the reviewers for their time and constructive feedback. We found the comments and suggestions of the reviewers extremely useful, which helped improve the manuscript substantially. Below we provide the summary of the rebuttal.

The main feedback and our responses

  • All the reviewers suggested more extensive adversarial evaluation to ensure that adversarial robustness is real. We conducted additional experiments that included longer step, multi start PGD attacks with different step sizes [SEwe W1 and Q1] as well as AutoAttack [Ca2d W2 and Q3, 111R W2 and Q3] which is constructed from white and black box attacks and confirmed the emergence of robustness in all the architectures we evaluated initially.
  • Reviewers SEwe and Ca2d asked about the impact of the suppression of massive activations on the adversarial accuracy and LC. We conducted a controlled suppression study on ViT, GPT, and Pythia 160M, measuring adversarial consistency (the proportion of predictions that remain unchanged under attack) and LC. Zeroing the largest activations reduced consistency and increased LC, whereas zeroing the same number of randomly selected activations left both metrics largely unchanged [SEwe W2 and Q3, Ca2d W2 and Q4]. These results illustrate that our proposed connection to local geometry, and prediction stability under adversarial perturbations is specific to massive activations.
  • Reviewers SEwe and Ca2d asked about the adequacy of LC metric for transformers. To validate LC as a proxy for local nonlinearity, we measured its correlation with the error of a first order approximation of the model around data points. We observed positive correlations across architectures and training stages, supporting the use of LC as an empirical indicator of departures from local linearity [SEwe W2 and Q3, Ca2d Q4]. Additionally, we would like to highlight that such LC proxy metrics were adopted in prior established work for non-max affine spline architectures (Humayun et al. 2024).
  • Reviewer Ca2d asked to what extent the theoretical assumptions are satisfied. We measured how often the dominance and non-cancellation assumptions hold in a residual ReLU MLP, which matches our theoretical setup, throughout training. Both assumptions and the bound from neuron hyperplanes increased during training [Ca2d W1 and Q2]. These results support the empirical relevance of the sufficient conditions.

Discussion highlights

  • After reviewing our responses, additional experiments and addressing the related work Reviewer 111R updated their score to 3, highlighting that our updates addressed their concerns supporting our study.
  • We have not yet received a response from Reviewers SEwe and Ca2d, but we hope that our new experiments and updates to the manuscript addressed their concern.

Impact and significance of the manuscript

We found a novel and unexpected connection between massive activations and adversarial robustness, through the change of local network geometry during training and showed this connection in a diverse mix of architectures and datasets, including LLMs. Our study provides a new perspective on the functional role of massive activations, connects concepts that were previously studied separately (namely massive activations and emergent adversarial robustness) and insights into and facilitates better understanding of the inner workings of neural networks.

Conclusion

We thank the reviewers for their valuable feedback, which led to substantial improvements of the manuscript. We hope that our responses, experiments and updates in the manuscript address the concerns from Reviewers SEwe and Ca2d and given that we have invested significant effort to address your comments we would highly appreciate their consideration to increase the score.

Best regards,

The authors

References

Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. “Deep Networks Always Grok and Here is Why.” ICML, 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.