- Rating
- 4(+1)
- Confidence
- 3(−1)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the mechanisms underlying massive activations (MAs) in large language models by analyzing 26 models using component ablations, singular value decomposition (SVD), and causal interventions. The authors identify the MLP down-projection as the primary source of MAs in most models and show that function-token representations preferentially align with the right-singular directions of the down-projection matrix. These aligned directions amplify and accumulate contributions at a small number of output coordinates, while attention primarily regulates this process. The authors further investigate both single-layer and multi-layer generation mechanisms and use targeted interventions to provide causal evidence for the role of the identified geometric structure. The results offer a unified explanation of MAs across a range of model architectures.
Critical strengths
-
The paper provides a systematic investigation of MA generation by connecting token-level triggers, layer-wise sources, and weight-space geometry. The combination of component ablations, frequency-controlled token analysis, SVD-based decomposition, and causal interventions offers a coherent approach to understanding the underlying mechanism.
-
The experiments cover 26 language models across multiple architectural families. The distinction between single-layer and multi-layer source regimes is particularly useful, and the causal interventions provide evidence beyond simple correlations between activation magnitudes and singular directions.
Critical weaknesses
-
The definition of function tokens combines several heterogeneous categories, including conventional function words, punctuation, structural markers, digits, and short BPE fragments. These categories may exhibit different activation patterns for different reasons. Consequently, the observed function-token enrichment may partly depend on the operational definition, making it difficult to determine how much of the effect is specifically attributable to linguistic function words rather than structural tokens or tokenization artifacts.
-
The causal interventions establish that perturbing the identified singular-vector geometry can substantially suppress MAs, but their specificity remains insufficiently characterized. Randomizing the right-singular-vector matrix changes the weight matrix globally, and removing selected directions may also affect other model computations. Without evaluating the impact on ordinary model behavior and comparing against carefully matched control interventions, it remains unclear how selectively these interventions target the MA-generating mechanism.
-
The main experiments use relatively short English text windows, and the independent-document robustness analysis covers only a subset of the evaluated models. Although the results are encouraging, their generalization to other corpora, languages, longer contexts, and different token distributions remains insufficiently established.
Questions for authors
-
The authors should incorporate and discuss the related work https://arxiv.org/abs/2510.10238. A more explicit comparison would help clarify the novelty of the proposed SVD-based geometric explanation relative to existing findings on critical neurons and MLP down-projection components. In particular, the authors should explain whether the identified right-singular directions provide a more specific mechanistic account of MA generation beyond identifying important neurons or layers.
-
The authors should conduct additional ablations to disentangle the contributions of different function-token categories. For example, conventional function words, punctuation and structural markers, digits, and short BPE fragments could be analyzed separately. Repeating the frequency-controlled analysis under these alternative definitions would help establish whether the reported token preference is robust and linguistically meaningful across model tokenizers.
-
The authors should strengthen the causal validation with matched control interventions, such as removing random directions with comparable dimensionality or norm, and compare targeted interventions against leading-direction ablations. They should also evaluate language-model perplexity and downstream task performance before and after intervention to determine whether MA suppression can be achieved without broadly disrupting model functionality.
2 comments
I appreciate the authors' thorough response. Their clarifications have completely resolved my initial concerns, and I am happy to increase my score accordingly.
Response on Function-Token Mechanisms, Performance Effects, and Generalization
Thank you for the review.
(RevC-1) Contribution beyond critical-neuron studies
Our work adds a quantitative connection between function-token (FT) representations, singular directions, actual MA-coordinate writes, and layer-dependence patterns. Qin et al.'s critical-neuron study identifies influential MLP components through intervention.
At each token, we measure norm, normalized alignment, projection, and output. The contribution of direction i to coordinate j is ci(t,j)=σi(vi⊤h2(t))ui[j]: vi determines input projection, σi amplification, and ui[j] output localization. We test low-order reconstruction against actual writes, including signed cancellation, bias, and higher-order contributions.
Across 26 models, per-layer/cumulative MLP ablations and recovery organize Concentrated (9), Few-Source (7), Dispersed (8), and Anomaly (2) patterns. Representative-model macro measurements additionally connect calibrated output directions to fixed-coordinate increments and actual layer writes. A concentrated direct writer can coexist with dependence on other layers. Directional interventions and matched controls connect this geometry to selective MA responses, yielding a quantitative taxonomy of MA activation patterns.
(RevC-2) Why combine different FT categories?
FT is the operational token category used to quantify the relation between token representations and MA writing. It combines conventional function words, punctuation/structural markers, digits, and short BPE fragments, including BOS.
The paired geometry experiments use Qwen2.5-7B-Instruct, Mistral-7B-v0.3, and Llama-3.1-8B-Instruct on 256 C4 documents of 256 tokens each. At the measured amplification blocks, FT accounts for 147/258, 503/503, and 256/256 strong events, respectively (same-token residual peak/median ratio ≥300). We measure norm, normalized direction, and actual coordinate writing separately; the observed model differences are retained. Our present focus is their quantitative correspondence and single-/multi-layer dependence. Systematic causal separation of FT subclasses and tokenizer effects remains future work.
(RevC-3) Can MA suppression and performance changes be distinguished?
MA-peak suppression and prediction-performance changes are separable under the tested interventions.
We tested 43 conditions in Q4 = Qwen3-4B, Q8 = Qwen3-8B, and L8 = Llama-3.1-8B-Instruct, using 128 C4 documents of 256 tokens/model. Directions and observation blocks are fixed before evaluation; we track the baseline token/coordinate.
First, removing 50% of the current target output-direction component gives:
Second, we compare target and orthogonal random directions at a separate 25% baseline-referenced activation budget. All five seeds match each intervened layer's aggregate actual output-change norm within 5%:
Random columns cover all five seeds. Random updates increase PPL while preserving most of the original peak; target updates suppress it.
We also tested full target-direction removal in the weights on 256 PIQA questions, scoring continuations by mean token log-likelihood. The 95% intervals use 2,000 paired question-bootstrap draws:
Complete removal has performance costs; the small-PPL results concern partial intervention.
(RevC-4) Transfer to other corpora and longer contexts
The C4-calibrated intervention transfers to WikiText and the tested 1,024-token windows, suppressing the original MA peak by over 99.8% in all three models.
We applied C4-calibrated directions, layer sets, and intervention weights directly to WikiText. We evaluated 128 documents at 256 tokens and paired the first 64 at 256 and 1,024 tokens. The table reports the complete short-window sample and the long-window subset:
Broader corpora, languages, context lengths, and token distributions are future work.