acceptodds
Under review as a conference paper at ICLR 2027

Massive Activation Gating Channel in Large Language Models

Abstract

Massive activations, a phenomenon in which a small number of hidden channels exhibit exceptionally large magnitudes, are pervasive in large language models (LLMs). However, the mechanism by which a token develops massive activations as it propagates through a pretrained LLM remains poorly understood. In this paper, we find that the emergence of massive activations is controlled by a single channel in the input embedding to a spike feed-forward network (FFN). The position of this channel is fixed for a particular LLM. We name this channel the massive activation gating channel (MAGC). When the value of the MAGC is sufficiently large (or small, depending on the LLM), the output of the spike FFN exhibits massive activations. Examining six LLMs across four model families and different model sizes, we verify the existence and effect of MAGC. We further provide a theoretical explanation of the mechanism by which MAGC induces massive activations. When the value of MAGC is sufficiently large (or small), the output of a spike FFN asymptotically reduces to a quadratic form that mixes a few columns of the down-projection matrix of the FFN. Since these columns exhibit the shape of massive activations, the output therefore exhibits massive activations.

Primary area:
generative models
Keywords:
massive activations, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KFDuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper mainly focuses on the formation mechanism of massive activations in LLMs. Specifically, this work identified a spatially fixed and direction-specific MAGC within the input to early-layer spike FFNs. When the value of this channel is sufficiently large or small, the SwiGLU FFN output asymptotically approaches a quadratic weighted combination of a few columns of the corresponding matrix. Furthermore, this work also validates this mechanism through random inputs, single-channel interventions, and theoretical derivations across six LLMs of varying scales and model families, further demonstrating that modulating the MAGC can significantly alter attention sinks.

Critical strengths
  • This work presents the MAGC mechanism and locates massive activations from empirical phenomena to parameter-level causal chains between single input channels, SwiGLU gating, and a handful of down-projection columns.

  • The theoretical analysis and experiments are well combined.

  • Consistent structures are observed on models with 1B-32B parameters. Besides, MAGC demonstrates potential application value via the attention sink intervention.

Critical weaknesses
  • The most comprehensive MAGC substitution experiments, coefficient concentration analyses, and output collinearity analyses focus primarily on LLaMA-3.2-1B. For the other models, only visualizations based on a limited number of random sequences are provided, without systematic reporting of intervention success rates, effect sizes, variance, or cross-dataset stability. As a result, the existing evidence is insufficient to support the central conclusion that MAGC is a universal mechanism across different model families.

  • This work locates the MAGC by analyzing the difference in average FFN inputs between the first and second tokens and identifies spike FFNs using a manually set threshold. However, it fails to demonstrate the robustness of these results regarding token selection, prompts, random seeds, or threshold values. Especially in the case of QwQ-32B, the authors had to skip several channels exhibiting the largest differences and manually select a channel that was not a massive-activation channel. This indicates that the identification rule does not consistently apply across all models and undermines the claim that a unique, fixed MAGC exists for every model.

  • Theorem 4.1 describes the general asymptotic behavior of an arbitrary input channel as x→±∞x \to \pm\infty. However, MAGC values ​​are finite in practice, e.g., approximately 8.8 for LLaMA-3.2-1B. The paper does not provide error bounds, approximation accuracy, or trigger thresholds for finite values ​​of xx, nor does it theoretically derive why the coefficients associated with MAGC are so highly concentrated. This key property is primarily established through empirical observation.

Questions for authors

Please refer to the Critical Weaknesses part.

2 comments
AuthorsOpenReview ↗

Response to Reviewer KFDu

W1 generalizability of results 

Thank you for your comments on the generalizability of MAGC on other models. The coefficient concentration analyses and output collinearity analyses are included in the corresponding subfigures for each model. For example, Figure A6(c) shows that the coefficient vector (α\alpha) is concentrated, while Figures A6(d)–(f) show the collinearity between the columns of the spike FFN and its output.

To evaluate the generalizability of the MAGC substitution experiments, we conduct the same experiments across different models. The results show that adjusting the MAGC of non-spike tokens in the corresponding gating direction consistently induces massive activations.

W2 stability and uniqueness of MAGC

Thank you for raising these important questions. We have further examined the stability of our MAGC identification procedure across different samples.

To improve the robustness of identification, we now calculate the channel statistics by averaging over 100 samples, rather than relying on a single sample. We find that, for all models except QwQ-32B, the top-1 channel identified by the argmax criterion is exactly the same across all 100 samples (100/100). This provides strong evidence that the identified MAGC is stable across different input samples for these small models. We acknowledge that this identification procedure is less robust for the larger QwQ-32B model. This limitation suggests that identifying MAGC in larger models may require further investigation, and we will clarify this point in the revised manuscript.

We also acknowledge that our experiments cannot rule out the possibility that multiple input channels jointly contribute to massive activations. Empirically, the results on the focal LLaMA-1B model provide evidence for the uniqueness of MAGC, as manipulating the identified channel can both induce and suppress MAs. However, for Qwen2.5B, manipulating a single channel can induce MAs in non-spike tokens, but cannot suppress MAs in existing spike tokens. This suggests that multiple input channels may contribute to the formation or maintenance of MAs.

Theoretically, if two input channels have similar or aligned FFN parameter structures, such that activating either channel leads to a concentration of α\alpha, both channels could potentially induce the same MA phenomenon. Therefore, our findings do not establish that MAGC is the only input channel capable of producing MAs. Rather, our experiments demonstrate that at least one identifiable input channel, i.e., the MAGC, can control the occurrence of MAs.

W3 empirical error of the asymptotic assumption 

Thank you for your comment. As suggested, we compared the actual spike-FFN output with the Theorem 4.1 approximation x2y∗x^2 y^*, for all six models.

Specifically, we feed 20 natural English sentences (without BOS token) to each model. For every spike token (the first token), we take its real input hh to the spike FFN and its real output yy, and let x=h(i∗)x = h(i ^* ) be the natural MAGC value. We compute the cosine similarity between yy and the Theorem 4.1 approximation $x^2 y^* $, where $y^* $ is computed from the weights alone.

ModelSpike tokensNon-spike tokens
LLaMA-3.2-1B0.9999−0.01
LLaMA-2-7B0.982−0.05
Qwen2.5-1.5B0.9940.04
QwQ-32B0.6420.14
Ministral-3-3B0.4990.13
DeepSeek-R1-8B0.9995−0.02

Table. Cosine similarity between the real output yy and the Theorem 4.1 approximation x2y∗x^2 y^* (for non-spike tokens, we compute the mean of all non-spike tokens).

In four of the six models (LLaMA-3.2-1B, LLaMA-2-7B, Qwen2.5-1.5B, and DeepSeek-R1-8B), the direction predicted by Theorem 4.1 from the pretrained weights closely matches the actual spike-FFN output at natural MAGC values, with cosine similarities above 0.98. In contrast, the outputs of non-spike tokens are nearly orthogonal to the predicted direction. For QwQ-32B and Ministral-3-3B, the cosine similarities are relatively lower (0.64 and 0.50, respectively), but remain substantially higher than those of non-spike tokens (0.14 and 0.13).

These results suggest that the asymptotic direction captures a component of the actual spike-FFN output, which is consistent with the asymptotic nature of Theorem 4.1.

Thank you for the response. I have decided to maintain my rating of 2. The following concerns still affect my assessment.

  • The cross-model intervention results still lack per-model sample counts, success rates, effect sizes, and variability across inputs. Consequently, it is difficult to assess how reliably the intervention induces massive activations across the evaluated models. The columns reported in Table 1 are selected partly for their alignment with the observed output, which also limits the value of the appendix plots as independent evidence for the proposed mechanism.

  • Identifying the MAGC in QwQ-32B still requires an additional filtering step among the ten highest-ranked channels. The sensitivity of the identification procedure to token choice and the spike-FFN thresholds remains unclear. In the Qwen experiment, the single-channel intervention failed to suppress existing massive activations. This leaves open whether a single fixed channel can account for naturally occurring massive activations across the evaluated models.

  • At natural MAGC values, the cosine similarities for QwQ-32B and Ministral-3-3B are 0.642 and 0.499, respectively, indicating limited directional agreement in these two models. Cosine similarity is insensitive to the x2x^2 scaling factor, so it leaves the accuracy of the predicted output magnitude untested, including on the dimensions that exhibit massive activations. The finite range over which the quadratic approximation accurately reproduces those activations therefore remains unclear.

Reviewer_z9q7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies where massive activations come from in decoder-only LLMs. The authors claim that each model has a single fixed channel that controls massive activations, and they call it the massive activation gating channel (MAGC).

Critical strengths

The authors explain the mechanism using only the weights, and their observations match the output patterns across six models. The MAGC can also control the attention sink, which could be useful for quantization and KV-cache control.

Critical weaknesses

Part of the mechanism appears to overlap substantially with two uncited works: The Super Weight in Large Language Models (M. Yu et al., 2024) and House of Cards: Massive Weights in LLMs (Oh et al., 2024).

There is also a gap between the theory and the operating regime. Theorem 4.1 holds only in the limit and applies to any channel of any SwiGLU FFN. The realistic MAGC value is moderate, but the paper does not quantify whether y ≈ x²y* holds.

Numerous papers have investigated massive activations and attention sinks. Therefore, this work should clearly differentiate itself from prior studies or demonstrate downstream use cases.

Questions for authors
  1. What is the FFN output norm as a function of c in Fig. 4? If massive activations persist at c = 0, what produces them?
  2. At realistic MAGC values, please quantify the relative error.
  3. How do i*, j*, and J relate to the super weights of M. Yu et al. (2024) and the massive weights of Oh et al. (2024)? What does this work explain beyond these works?
  4. A question frequently raised in attention sink research: does the MAGC also operate on tokens other than the first token, such as newline tokens?
5 comments
AuthorsOpenReview ↗

Response to Reviewer z9q7

Weakness 2 There is also a gap between the theory and the operating regime. Theorem 4.1 holds only in the limit and applies to any channel of any SwiGLU FFN. The realistic MAGC value is moderate, but the paper does not quantify whether y ≈ x²y* holds.
Question 2 At realistic MAGC values, please quantify the relative error.

Thank you for your comment. As suggested, we compared the actual spike-FFN output with the Theorem 4.1 approximation x2y∗x^2 y^*, for all six models.

Specifically, we feed 20 natural English sentences (without BOS token) to each model. For every spike token (the first token), we take its real input hh to the spike FFN and its real output yy, and let x=h(i∗)x = h(i^* ) be the natural MAGC value. We compute the cosine similarity between yy and the Theorem 4.1 approximation $x^2 y^* $, where $y^* $ is computed from the weights alone.

ModelSpike tokensNon-spike tokens
LLaMA-3.2-1B0.9999−0.01
LLaMA-2-7B0.982−0.05
Qwen2.5-1.5B0.9940.04
QwQ-32B0.6420.14
Ministral-3-3B0.4990.13
DeepSeek-R1-8B0.9995−0.02

Table. Cosine similarity between the real output yy and the Theorem 4.1 approximation x2y∗x^2 y^* (for non-spike tokens, we compute the mean of all non-spike tokens).

In four of the six models (LLaMA-3.2-1B, LLaMA-2-7B, Qwen2.5-1.5B, and DeepSeek-R1-8B), the direction predicted by Theorem 4.1 from the pretrained weights closely matches the actual spike-FFN output at natural MAGC values, with cosine similarities above 0.98. In contrast, the outputs of non-spike tokens are nearly orthogonal to the predicted direction. For QwQ-32B and Ministral-3-3B, the cosine similarities are relatively lower (0.64 and 0.50, respectively), but remain substantially higher than those of non-spike tokens (0.14 and 0.13).

These results suggest that the asymptotic direction captures a component of the actual spike-FFN output, which is consistent with the asymptotic nature of Theorem 4.1.

Weakness 1 Part of the mechanism appears to overlap substantially with two uncited works: The Super Weight in Large Language Models (M. Yu et al., 2024) and House of Cards: Massive Weights in LLMs (Oh et al., 2024).
Weakness 3 Numerous papers have investigated massive activations and attention sinks. Therefore, this work should clearly differentiate itself from prior studies or demonstrate downstream use cases.
Question 3 How do i*, j*, and J relate to the super weights of M. Yu et al. (2024) and the massive weights of Oh et al. (2024)? What does this work explain beyond these works?

Thank you for your question. We acknowledge the overlap between our findings and these prior studies. However,our work also makes distinct contributions, as we explain below. We will cite and discuss these studies in the revised manuscript.

Relationship to prior findings. Our findings are consistent with the important weight structures identified in prior studies.

Specifically, Yu et al. (2024) identified super weights in the FFN down-projection matrix, while Oh et al. (2024) identified massive weights associated with large intermediate FFN activations. For example, in LLaMA-2-7B, Yu et al. identified a super weight at (Wdown[2533,7890])(W_{\mathrm{down}}[2533,7890]), while Oh et al. reported 7890 and 10411 among their top-five massive-weight indices. Consistently, our work identifies (J={10411,7890}) as the dominant down-projection columns. These findings establish a connection between our identified columns and the weight structures reported in prior studies.

Our contributions beyond prior work. Compared with these studies, our work makes two additional contributions:

  1. Explaining token-dependent massive activations. Previous studies have identified important weights associated with massive activations. However, since these weights are fixed across tokens, why do massive activations emerge in some tokens but not others? We identify MAGC as a key input channel whose value determines whether massive activations emerge, providing an explanation for this token-dependent behavior.
  2. Providing a theoretical explanation of the gating mechanism. We further establish a mathematical connection between MAGC and the resulting massive activations. Specifically, Theorem 4.1 shows how a large MAGC value interacts with the gate and up-projection matrices, causing the FFN output to approach a quadratic combination of selected down-projection columns. This explains how the input MAGC value activates the weight structures associated with massive activations.
Question 1 What is the FFN output norm as a function of c in Fig. 4? If massive activations persist at c = 0, what produces them?

Thank you for your question.

1. FFN output norm as a function of (c). We examined the spike-FFN output norm as a function of (c) and observed a trend similar to the attention sink score in Fig. 4. We will include this result in the revised manuscript.

2. Why massive activations persist at (c=0). The gating behavior depends on the combined contributions of MAGC and the other input channels. Specifically, the gate pre-activation of an intermediate channel (j) can be written as wg,jc+bg,jw_{g,j}c+b_{g,j}, where bg,jb_{g,j} represents the contribution from the remaining input channels. Therefore, the gate can remain activated at (c=0) when bg,j>0b_{g,j}>0, and becomes suppressed only when (c) falls below a threshold determined by these contributions (assuming wg,j>0w_{g,j}>0). This explains why massive activations can persist at (c=0) but disappear when (c) becomes sufficiently negative.

This is consistent with Theorem 4.1, which characterizes the asymptotic behavior for large MAGC values rather than requiring massive activations to vanish at (c=0).

Question 4 A question frequently raised in attention sink research: does the MAGC also operate on tokens other than the first token, such as newline tokens?

Thank you for your question. MAGC is not specific to the first token. It is a fixed input channel of the spike FFN, and the same gating mechanism applies to tokens at other positions. Whether a token develops massive activations depends on its MAGC value and the contributions from the remaining input channels, rather than its position in the sequence.

Therefore, tokens other than the first token, including newline tokens, can potentially trigger massive activations through MAGC. We further conducted experiments on newline tokens and confirmed that MAGC also operates on these tokens. Specifically, newline tokens with sufficiently large MAGC values can develop massive activations through the same gating mechanism. We will include these results in the revised manuscript.

Thank you for the response.

I appreciate that you acknowledge the overlap with Yu et al. and Oh et al. and will cite them. But since J is the same as the indices reported in those works, the new part of this paper is the claim that MAGC decides which tokens spike. I don't think the rebuttal supports this claim yet.

  1. I asked for the relative error, but the table reports cosine similarity only. Also, QwQ-32B and Ministral-3-3B are at 0.64 and 0.50, so the theorem does not seem to describe these two models.
  2. Q1 says the spike remains at c = 0 because other channels keep the gate open. Then MAGC is not required for the spike.
  3. For newline tokens, I want to know if the tokens that spike naturally have large MAGC values without intervention.
  4. Downstream use cases were not answered.

I will keep my rating for now.

Reviewer_mDX7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the cause of massive activations in LLMs. It identifies a specific input channel in an early FFN, called the Massive Activation Gating Channel (MAGC), which is argued to govern the emergence of massive activations. The authors theoretically show that as this channel's magnitude grows toward infinity in the correct direction, the FFN output approximates a quadratic combination of selected down-projection columns. The proposed mechanism is examined across six LLMs from four families (1B–32B parameters), with intervention experiments primarily conducted on LLaMA-3.2-1B. Modifying MAGC is also shown to affect attention sink behavior in the following transformer block.

Critical strengths

S1. An interesting and interpretable mechanism for massive activations.

Identifying MAGC provides a concrete, channel-level perspective on how massive activations arise in pretrained LLMs. Instead of simply describing activation outliers, the paper tries to account for their origin through specific interactions between FFN inputs and weight matrices. This represents a valuable step toward understanding the inner workings of LLMs.

S2. A theoretically grounded explanation supported by evidence across models.

The asymptotic analysis links the proposed gating channel to a weighted combination of columns in the FFN down projection matrix. The authors examine six models spanning different families and sizes, offering initial evidence that the identified mechanism generalizes beyond a single architecture. Combining theoretical reasoning with empirical findings strengthens the paper's potential contribution.

Critical weaknesses

W1. Insufficient evidence that the asymptotic analysis explains massive activations under realistic inputs.

Theorem 4.1 describes FFN output when one input channel tends to infinity, but massive activations occur at finite values during normal inference, e.g., MAGC around 8.8 in LLaMA-3.2-1B. The paper does not quantify how accurate the asymptotic approximation is at such realistic values, lacking direct comparisons like reconstruction error, cosine similarity, or magnitude ratio between actual FFN output and the approximation. Thus, whether the decomposition faithfully explains actual massive activation generation remains unclear, affecting the core mechanistic claim.

W2. Insufficient causal validation of the single-channel control mechanism.

The paper claims that a single MAGC controls massive activations and that its position is fixed for each model. However, the main experiments directly examining massive activation generation rely on synthetic Pareto-distributed inputs in LLaMA-3.2-1B, with only a few alternative channels presented as controls. Although Section 5.2 includes interventions on natural sentences, these experiments primarily evaluate attention sink behavior in a single model. The paper lacks systematic causal comparisons across models and input distributions to establish the necessity, uniqueness, and robustness of MAGC. Validation on the remaining models relies mainly on channel identification, weight analysis, and visualizations rather than comparable interventions. Consequently, the evidence does not fully establish MAGC as a unique and general control mechanism.

Questions for authors

Q1 (Regarding W1). Accuracy of the asymptotic approximation under realistic inputs.​

Can the authors quantitatively compare the actual spike FFN output with the Theorem 4.1 approximation using naturally occurring MAGC values? Reporting reconstruction errors, cosine similarities, and magnitude ratios across all six models would clarify whether the analysis explains observed activations or only limiting behavior.

Q2 (Regarding W2). Necessity and sufficiency of MAGC across models.​

Can the authors systematically quantify the effects of MAGC suppression and enhancement on massive activation magnitudes using natural inputs across all six models? In particular, comparisons with matched-magnitude interventions on other channels would help establish whether MAGC is uniquely responsible for the observed behavior, beyond the single-model experiments reported in Sections 3.2 and 5.2.

Q3 (Regarding W2). Stability and uniqueness of MAGC across inputs.​

Is the identified MAGC stable across different prompts and sequence lengths? In QwQ-32B, Appendix D introduces a modified identification procedure that excludes channels associated with massive activations propagated through the residual stream. How robust is the identified MAGC to this selection procedure, and could multiple input channels contribute to the observed phenomenon?

1 comment
AuthorsOpenReview ↗

Response to Reviewer mDX7

W1. Insufficient evidence that the asymptotic analysis explains massive activations under realistic inputs.
Q1. Accuracy of the asymptotic approximation under realistic inputs.

Thank you for your comment. As suggested, we compared the actual spike-FFN output with the Theorem 4.1 approximation x2y∗x^2 y^*, for all six models.

Specifically, We feed 20 natural English sentences (without BOS token) to each model. For every spike token (the first token), we take its real input hh to the spike FFN and its real output yy, and let x=h(i∗)x = h(i^* ) be the natural MAGC value. We compute the cosine similarity between yy and the Theorem 4.1 approximation $x^2 y^* $, where $y^* $ is computed from the weights alone.

ModelSpike tokensNon-spike tokens
LLaMA-3.2-1B0.9999−0.01
LLaMA-2-7B0.982−0.05
Qwen2.5-1.5B0.9940.04
QwQ-32B0.6420.14
Ministral-3-3B0.4990.13
DeepSeek-R1-8B0.9995−0.02

Table. Cosine similarity between the real output yy and the Theorem 4.1 approximation x2y∗x^2 y^* (for non-spike tokens, we compute the mean of all non-spike tokens).

In four of the six models (LLaMA-3.2-1B, LLaMA-2-7B, Qwen2.5-1.5B, and DeepSeek-R1-8B), the direction predicted by Theorem 4.1 from the pretrained weights closely matches the actual spike-FFN output at natural MAGC values, with cosine similarities above 0.98. In contrast, the outputs of non-spike tokens are nearly orthogonal to the predicted direction. For QwQ-32B and Ministral-3-3B, the cosine similarities are relatively lower (0.64 and 0.50, respectively), but remain substantially higher than those of non-spike tokens (0.14 and 0.13).

These results suggest that the asymptotic direction captures a component of the actual spike-FFN output, which is consistent with the asymptotic nature of Theorem 4.1.

W2. Insufficient causal validation of the single-channel control mechanism.
Q2. Necessity and sufficiency of MAGC across models.

Following your comment, we conducted additional intervention experiments using 10 natural-language sentences, initially focusing on LLaMA-3.1-1B. Specifically, we performed three sets of experiments:

  1. MAGC suppression on spike tokens. We suppressed the MAGC value of spike tokens by setting it to -100. We observed that the massive activation (MA) disappeared, providing evidence for the necessity of MAGC in generating MA.

  2. MAGC enhancement on non-spike tokens. We increased the MAGC value of non-spike tokens to 100. Interestingly, these tokens became spike tokens, suggesting that enhancing MAGC is sufficient to induce MA under the tested conditions.

  3. Interventions on other channels. We selected several other random channels and varied their values from -100 to 100 for non-spike tokens. None of these interventions independently induced MA, supporting the specificity of MAGC's role.

Furthermore, we extended these intervention experiments to the other models. We find the MAGC is not unique outside the LLaMA model, which we will discuss in the next comment.

Q3. Stability and uniqueness of MAGC across inputs.

Thank you for raising these important questions. We have further examined the stability of our MAGC identification procedure across different samples.

To improve the robustness of identification, we now calculate the channel statistics by averaging over 100 samples, rather than relying on a single sample. We find that, for all models except QwQ-32B, the top-1 channel identified by the argmax criterion is exactly the same across all 100 samples (100/100). This provides strong evidence that the identified MAGC is stable across different input samples for these small models. We acknowledge that this identification procedure is less robust for the larger QwQ-32B model. This limitation suggests that identifying MAGC in larger models may require further investigation, and we will clarify this point in the revised manuscript.

We also acknowledge that our experiments cannot rule out the possibility that multiple input channels jointly contribute to massive activations. Empirically, the results on the focal LLaMA-1B model provide evidence for the uniqueness of MAGC, as manipulating the identified channel can both induce and suppress MAs. However, for Qwen2.5B, manipulating a single channel can induce MAs in non-spike tokens, but cannot suppress MAs in existing spike tokens. This suggests that multiple input channels may contribute to the formation or maintenance of MAs.

Theoretically, if two input channels have similar or aligned FFN parameter structures, such that activating either channel leads to a concentration of α\alpha, both channels could potentially induce the same MA phenomenon. Therefore, our findings do not establish that MAGC is the only input channel capable of producing MAs. Rather, our experiments demonstrate that at least one identifiable input channel, i.e., the MAGC, can control the occurrence of MAs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.