acceptodds
Under review as a conference paper at ICLR 2027

The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation

Abstract

Massive activations, residual-stream coordinates with magnitudes far larger than typical activations, are associated with attention sinks in transformers, but how their scale is regulated during training remains incompletely understood. Combining training-trajectory analyses and controlled interventions, we trace their emergence, growth, and consolidation. Sink-carrying channels vary across random seeds but stabilize early within each run. Over longer training, surrounding channels erode and the sink concentrates onto a few redundant carriers. Across ablations, gradient attenuation follows the sink token’s collective root-mean-square magnitude rather than any single channel, making collective scale central to understanding their effects. Our central result is that weight decay causally controls the turnover of global activation scale. In controlled continuations, removing decay near the peak allows this scale to keep rising, whereas retaining it produces decline even at constant learning rate. We develop a balance model for the rise and peak of massive-activation magnitude, in which AdamW-preconditioned growth opposes weight decay. Sweeping the decay coefficient shifts peak timing approximately log-linearly and yields peak magnitudes scaling approximately as , consistent with this balance. Optimizer measurements further show that preconditioning sustains the large-channel cohort against decay even when raw maintaining forces are too small to do so. Together, these findings connect the observed life cycle to scale-regulating training dynamics and establish weight decay as a training-time lever on activation magnitude.

Primary area:
foundation or frontier models, including LLMs
Keywords:
weight decay, training dynamics, massive activations, attention sinks, AdamW, activation-scale regulation, transformers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_q2apOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the emergence, growth, and concentration of massive residual-stream activations. It combines training runs of 51M–380M transformers with analyses of released Pythia checkpoints spanning 70M–6.9B parameters. Its main findings concern early, seed-dependent channel identity; collective RMS as a relevant scale for gradient attenuation; weight decay’s effect on global activation turnover; and increasing concentration onto a small channel cohort. Validation includes checkpoint analyses, channel ablations, perturb-and-restore experiments, learning-rate controls, and a four-point weight-decay sweep. A balance model connects activation peaks to optimizer-mediated weight growth and decay, while survivor selection remains unresolved.

Critical strengths
  1. Preserving optimizer state and batch order while changing decay, together with a constant-learning-rate control, helps establish that the global median’s decline cannot be attributed solely to the learning-rate schedule.

  2. Separating raw magnitude, max-to-median ratio, and token RMS prevents misleading interpretations of activation trajectories. The matched-RMS perturbations also improve on random-channel comparisons by addressing a concrete magnitude confound. These controls make the observations in the 51M continuum regime more informative.

Critical weaknesses
  1. Equation (7) is a stationarity condition for the writer-weight norm. Applying it to an activation peak requires a sufficiently stable relationship between activation magnitude and that norm. Yet Appendix A reports that the activation declines while its writer norm continues increasing: the norm rises from 10.8 to 11.5 while activation per unit norm falls from 4.2 to 3.2. The changing relationship between weights and activations is therefore central to the phenomenon, and its cause remains unexplained. Furthermore, the drive exponent q=−1q=-1 is inferred from the measured peak-scaling relationship; this is model calibration, not an independent prediction of that relationship. Section 4.2 also explicitly limits the clean turnover intervention to the global median, with the individual massive activation plateauing under constant learning rate. These results support an effect of decay on activation scale, but leave a substantial gap in the proposed explanation of the massive activation’s peak.

  2. Appendix E estimates a preconditioned gradient from fresh gradient statistics on WikiText-2 at fixed Pythia checkpoints. This is not the historical AdamW update, whose first and second moments depend on the preceding training trajectory. The paper does not establish that this proxy preserves the sign or magnitude of the actual maintaining force, yet compares it directly with the decay coefficient and concludes that AdamW preconditioning sustains the cohort. That inference exceeds the evidence. The overlapping forces for winners and collapsers further leave the proposed “competitive consolidation” without an identified selection mechanism. The larger-model observations cannot resolve these gaps because the reported perturb-and-restore experiments are confined to the 51M continuum regime. Validating these claims requires substantive evidence about actual optimizer dynamics.

Questions for authors
  1. What evidence validates the WikiText-2-based maintaining-force proxy against actual AdamW updates during training, particularly its sign and its position relative to the decay threshold? Are such comparisons available for the models whose optimizer states were saved?

  2. At the measured activation peak, does the actual preconditioned maintaining force satisfy Equation (7), and does the writer-weight norm simultaneously become stationary? How is the claimed activation-peak condition reconciled with the continued growth of that norm reported in Appendix A?

  3. Appendix E describes the sweep as repeating a near-peak continuation protocol, whereas Figure 1b reports peak locations ranging from approximately 4k to 17k steps. Were these runs trained from initialization at each decay coefficient, or continued from checkpoints? Please specify the fork steps, observation windows, and whether the plotted steps are absolute or relative. This directly affects the interpretation of the scaling evidence.

1 comment
AuthorsOpenReview ↗

Review responses

thank you for your detailed analysis; you helped us identify several issues in the original analysis that upon correction made the paper much stronger. We have also corrected some relevant measurements and repeated some experiments where needed:

-eq. 7 and continued writer-norm growth: Your observation exposed a measurement error in our original analysis. The reported values were taken at layer 2, but the aggregate writer norm included blocks that could not contribute to that layer. We have corrected this by measuring at the MA's own layer (5), using the saved AdamW moments (Table 2). With this correction, the windowed maintaining force predicts the observed writer-norm growth to within 0.025 and crosses λ\lambda near step 21k, where the writer norm peaks. The MA itself peaks at 20k. In our second seed, both peak at 19k. We also find that m/∥w∥m/\|w\| remains stable within 2% after step 14k. These results show that Eq. 7 does locate the writer-norm peak. We have withdrawn the earlier claim that the decline is not due to weight shrinkage and corrected Appendix A accordingly.

-calibrated exponent: We agree that the exponent qq should not have been presented as an independent prediction. It is now explicitly identified as a calibrated parameter. The independent evidence for the proposed mechanism comes from the direct optimizer-state measurements described above.

-the proxy: We tested the proposed proxy against the actual optimizer dynamics, as you suggested. It failed to predict both the sign and magnitude of the observed changes, so we have withdrawn it (General Response, Point 1). Direct measurements also show that winning, collapsing, and randomly selected channels grow at nearly the same rate. This changes our interpretation: preconditioning appears to maintain the scale of the channel population rather than selectively supporting a particular cohort.

-constant learning rate: As discussed in our response to reviewer yjgi, the revised experiments distinguish the behavior of the MA from the global activation scale. Weight decay is necessary for the observed MA turnover, although under a constant learning rate the MA approximately plateaus rather than declining.

-sweep protocol: We identified an issue in our description of the original sweep. The runs were forked at step 4k, not near the peak as originally stated, which censored the result for λ=0.4\lambda=0.4. We have now repeated the full sweep for all four values of λ\lambda, starting from step 2k (Fig. 1b, Fig. 5b, and Appendix G). The corrected results show a strong negative correlation between λ\lambda and peak timing (−0.99 for the MA and −1.00 for the writer norm). The peak writer norm scales approximately as λ−0.60\lambda^{-0.60}, reasonably close to the predicted λ−1/2\lambda^{-1/2}. The MA magnitude follows a steeper scaling of approximately λ−0.76\lambda^{-0.76}, consistent with Eq. 7 constraining the writer norm ∥w∥\|w\|, rather than the activation magnitude mm directly.

perturb-and-restore at 51M: We agree that this remains a limitation and have made it explicit in §4.5. Magnitude-matched controls require other channels of comparable magnitude, which are available only in the continuum regime. We therefore do not claim that this experiment establishes the same behavior across larger models.

Again, thank you for taking the time to give such detailed feedback, it was very helpful in improving the results

Reviewer_yjgiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

This paper studies how massive activations appear and concentrate in a small number of residual-stream channels during training. It combines transformers trained from scratch at 51M, 203M, and 380M parameters with analyses of released 70M-6.9B parameter Pythia checkpoints. The experiments include tracking activations during training, removing selected channels, and resuming training under different settings. The authors report that channel identities stabilize early and that weight decay affects whether the global activation scale rises or falls.


Message to authors

I believe the paper provides an interesting empirical characterization of massive activations and several well-designed controls. However, I the strongest mechanistic claims are not sufficiently supported by the current evidence. In particular, I would like to see more evidence regarding the optimizer-driven maintenance and RMS-dependent gradient attenuation. I believe that addressing these could potentially strengthen the claims made in the paper.

Critical strengths
  1. The weight-decay experiment is a good ablation. The global median activation scale still falls with weight decay when the learning rate is held constant. This shows that a decaying learning-rate schedule is not necessary for this result. Keeping the starting checkpoint, optimizer state, and batch order the same strengthens the comparison. However, it does not fully establish the mechanism governing massive-activation decline (See Critical Weakness Minor 3)

  2. The distinction between massive-activation magnitude from global activation statistics. Methodologically, it is important to separate the max-to-median ratio, individual channel magnitude, and token RMS, while the analysis helps demonstrate why these quantities are not interchangeable.

  3. The experimental analysis goes beyond final checkpoints. Figure 2b is an important control for the magnitude-matched perturbations.

Critical weaknesses

Major Critical Weaknesses

  1. The AdamW analysis uses estimates rather than the original optimizer state. The text describes AdamW’s gradient averages using 200 WikiText-2 batches. These can differ from the optimizer values used during training, as they are computed on new batches at a fixed checkpoint. Without checking how close these estimates are to the actual values, it is not convincing that this establishes that AdamW maintains the large channels, as it is claimed in the manuscript.
  2. The gradient claim needs a stronger test. Equation 5 gives an upper bound on the gradient magnitude, but this does not necessarily mean the gradient is proportional to 1/RMS. Also, removing the massive activation changes RMS by only about 5%, so the experiment tests a small change. A larger range of RMS changes would help support this conclusion. Additionally, the gradient is measured at the block output, but Equation 5 concerns the gradient propagated through normalization. It is unclear whether these quantities are directly comparable.

Minor Critical Weaknesses

  1. The number of repeated runs is not clear. The setup lists two seeds as primary and second seed, but it is unclear how many independent runs support Figures 1 and 2. Please report the number for each experiment and show how much the results vary across runs.
  2. The conclusion makes a claim that was not tested. Lines 438-441 suggest that if a sink is suppressed, it will appear in other channels. However, the experiments only show that different seeds produce different sink channels. This does not prove that a suppressed sink will return. The authors also acknowledge this limitation in lines 449-451.
  3. Causal weight-decay result does not fully establish the mechanism governing massive-activation decline. Section 4.2 acknowledges it. If we apply a constant learning rate, then the global median activation declines and the individual massive activation plateaus. Could the authors distinguish it more it carefulIy?
Questions for authors
  1. Can you check the AdamW estimates against the actual optimizer state? This could be done with your own training runs. Do the actual values support the same conclusion about maintaining the large channels?
  2. Is the gradient measured at the block output the same gradient covered by Equation 5? Does the relationship still hold when RMS changes more substantially? Please also report the raw RMS and gradient values behind Figure 2a.
1 comment
AuthorsOpenReview ↗

Review responses

thank you for your detailed analysis; both of your major suggestions caused us to run some additional experiments that made the paper's main claim considerably stronger. We have updated the paper submission, and you can see our responses to your comments below:

  1. actual optimizer state:Following your suggestion, we now measure the maintaining force directly from the saved AdamW moments (General Response, Point 1). This measurement predicts writer-norm growth across individual windows and crosses λ\lambda at the writer-norm peak, providing direct evidence for the proposed weight-decay balance. We also found that our previous fresh-batch estimate was not a reliable substitute for the actual optimizer state, so we have replaced it and removed the cohort-level interpretation that depended on it.

  2. gradient measurements and Eq. 5:You were correct that the original Fig. 2a showed the total gradient, including the skip path, rather than the normalization-branch gradient bounded by Eq. 5. We now report both gradients alongside the upstream gradient gyg_y, over a wider 1.5–2× RMS range. The raw measurements are provided in Appendix E. In our primary seed, the normalization-branch gradient closely follows the predicted 1/RMS1/\mathrm{RMS} scaling, with slopes of −1.08, −0.99, and −1.40, while gyg_y remains nearly constant. Additional experiments with a second seed and Pythia confirm that the attenuation factor generalizes, although the stability of gyg_y depends on the run. We have revised §4.3 to make this distinction explicit (General Response, Point 2).

  3. runs and variability: We added Table 7 to identify the runs supporting each result. We also replicated the optimizer-state balance and gradient-floor analysis using a second seed, providing an additional check on the main findings.

  4. conclusion: We have revised the conclusion to present reconstitution as a hypothesis for future investigation, rather than an established result.

  5. constant learning rate: We now distinguish the behavior of the massive activation from the global activation scale in §4.1. Without weight decay, the massive activation continues growing (+43%). With weight decay, it declines under the learning-rate schedule (−17%) and approximately plateaus under constant learning rate (−2%). Meanwhile, the global median decreases in both settings. These results support the conclusion that weight decay is necessary for the observed turnover of the massive activation.

thank you again for the suggestions, particularly the recommendation to examine the actual optimizer state. That experiment strengthened the evidence for our central mechanism while also helping us identify and correct an interpretation that was not supported by the direct measurements.

Reviewer_8eWeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper ablates the life cycle of a massive activation (largest channel here) of a large language model. They experiment with a small Pythia model and observe the growth of that activation over time. They also observe global activation scale (median of channel) over time to oversee how things change in scale.

Critical strengths

Good description, first going over theory then going over results. I liked section 3.

I like that this paper has multiple findings about the life cycle of the massive activations.

Weight decay determining global activation scale is a very interesting finding. Also amount of weight decay determining when massive activation peaks is very interesting.

THE MASSIVE ACTIVATION IS A STOCHASTIC ROLE, NOT A FIXED DIMENSION is a intuitive but interesting observation. What about the number of massive activations? Does it change from training to training?

Section 4.4 which says (AFAIK) making a small activation large will pull it back to original place is an interesting finding.

Section 4.6 shows model grows larger the massive activations becomes more prominent. It matches the papers in quantization field that have seen this before [1]

There are many ablations done with carefully controlling the settings.

I like that this paper is honest about its limitatin that is only done on a small language model.

[1] https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/

Critical weaknesses

It feels the paper is trying to be too compact and I had to get help from AI to understand some of the sections. For example section 4.4 starts with "In perturb-and-restore experiments" where perturb and restore experiments are not defined before.

MA is never defined and can mean median activation (line 243), massive activation (Figure 1.b), please be consistent with it and define it early on in the paper. It looks like most of the time it is used as massive activation.

THE GRADIENT FLOOR IS ON THE TOKEN’S COLLECTIVE RMS --> This might not be true if the model is larger and ratio of most massive activation to second one is large. You can get a large model and do one forward backward pass to analyze it.

Questions for authors

Question: Why sink token matters when observing activation channels? Looks orthogonal to me.

Question: What is the key message you want to deliver? I understand it is about the life-cycle ofthe activation and multiple aspects are observed, but what is the most important one?

Question: Which result out of ablations is you think the most important? Put them first, section 4.1 referring to Appendix looks bad. I guess 4.2 should be there first.

Question: Why you don't use the full 9 pages and took some of the stuff to Appendix? The ethics could stay out of 9pages I guess.

Question: I am not familiar with the field very much, but isn't the fact that they follow a global low already studied in Gallego-Feliciano et al. (2025) as you have referenced in equation(2) yourself?

1 comment
AuthorsOpenReview ↗

Rebuttal responses

Thanks so much for your detailed review, it was very helpful. We have responded to each comment below, and made some edits in the main paper based on your feedback.

  1. clarity: We now define "MA" in the introduction as the largest channel at the sink token of the max-ratio layer. We also clarify the perturb-and-restore procedure when it is first introduced in §4.4.

  2. models with large top-1/top-2 ratios: We ran the suggested ablation on Pythia-1.4B and 2.8B, and your concern was well-founded. Removing the leading channel increases gyg_y by approximately 100× and 53×, respectively, so g⋅RMSg \cdot \mathrm{RMS} is not constant in these cases. A second seed showed the same effect, though less pronounced. We have revised §4.3 accordingly, narrowing the claim to what holds across all tested models: normalization-induced gradient attenuation depends on the collective RMS, rather than any single channel.

  3. number of MAs across runs: We observe between one and three sink channels, depending on the random seed (Table 1). Across five seeds each of Pythia-70m and 160m, the number of surviving channels does not appear to track sink attention strength (§4.6).

  4. why the sink token: We focus on the sink token because this is where the massive activations occur. The attenuation mechanism depends on that token's own RMS, while channels at other sequence positions do not reach comparable magnitudes.

  5. key message and organization: We have revised the end of the introduction to state the central finding more clearly: weight decay controls activation-scale turnover through its balance with AdamW-driven growth, while the particular channels that sustain the sink depend on the random seed. We also added a contribution list and moved the weight-decay results to the beginning of §4 to make the main finding more prominent.

  6. relation to Gallego-Feliciano et al. (2025): We have clarified how our work extends this earlier result. While Gallego-Feliciano et al. characterize the turnover dynamics, our experiments investigate their underlying cause. In particular, removing weight decay eliminates turnover, changing λ\lambda shifts its behavior, and examining optimizer state helps identify the mechanism responsible.

We appreciate these suggestions. In particular, the larger-model ablation revealed an important limitation in our original interpretation, and we believe the revised discussion is more precise as a result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.