acceptodds
Under review as a conference paper at ICLR 2027

Which the Eye Fears: Writing with Read-Blindness Explains Massive Activations in Transformers

Abstract

Massive activation features (MAs) in Transformers are extreme-value residualstream features that persist across layers despite the model’s ability to suppress them. Why do they survive? Our investigation using an operator-level mechanistic analysis of attention and feed-forward (FFN) blocks reveals that these blocks systematically ignore MA coordinates while reading, but not while writing; creating a read-write asymmetry that blocks corrective feedback while allowing continued accumulation. We find that both attention and feed-forward layers have this read-blindness, and contribute to the emergence and persistence of MAs. To validate prior work that hypothesized that FFN’s amplification abilities is the primary reason for MAs (Sun et al., 2026), we analyze the model checkpoints during learning. Contrary to our expectation, read-blindness emerges before FFN amplification, suggesting that it acts upstream in the MA mechanism. We further contribute gradient analysis to link this behavior to surprising asymmetries in the loss landscape, concluding that the model actively maintains this read-blindness. Finally, we find that removing read-blocking at different locations induces compensatory shifts elsewhere, but MAs still persist.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
massive activations, transformers, mechanistic interpretation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xNijOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper attributes the persistence of massive activations to suppressed reading alongside open writing. It studies weight Grams, activation second moments, orthogonal or frozen training interventions, an FFN quadratic approximation, checkpoint timing, and local AdamW updates. MA coordinates have lower relative read-side weight energy while still receiving updates, and constraining one read pathway does not eliminate MAs.

Critical strengths
  1. Decouple structural weights, input availability, actual update energy, and the sign of updates to avoid conflating non-zero writes with accumulation in the same direction.
  2. Parameterizing WVW_V via Cayley orthogonality preserves learning capacity more effectively than simple freezing; with PPL shifting only from 14.21 to 14.24, this represents a structurally meaningful intervention.
Critical weaknesses

This paper highlights an intriguing phenomenon: "massive activation" arises from an asymmetry between the null spaces of the input and output projections—an empirical observation that has largely been overlooked. This, however, also points to the paper's limitations.

In essence, the paper provides an empirical demonstration that these "massive values" are not read out by WOW_O or WdownW_{down}. Other claims—such as the assertion that massive activation represents an optimization fixed point—lack rigorous analysis. Consequently, I find the paper to be somewhat lacking in depth.

Questions for authors

See weakness

4 comments

We respectfully disagree that the paper's contribution is limited to observing that massive coordinates are weakly read by input projections. The paper also investigates this asymmetry through training dynamics, structural interventions, and local optimization analysis.

We point to Section 6 in the paper that discusses optimization interplay with read-write asymmetry. Our directional-Hessian analysis finds higher curvature along massive-coordinate directions in read-side matrices, whereas the corresponding write-side directions exhibit lower curvature. This identifies a local loss geometry in which the read-side configuration are being held tightly. Complementing this, our AdamW update analysis finds positive first-order pressure on massive activation magnitude across layers, driven primarily by the optimizer’s accumulated moments. Combined, these results indicate that the AdamW optimizer deliberately maintain the read-blindness and the massive activations during the training.

Further, we also conduct intervention analysis in Section 4.2 where we constrains read-side matrices through frozen orthonormal initialization or Cayley reparameterization. These interventions alter massive activation magnitudes and reveal compensation through other pathways. Thus, we tests the consequences of restricting the proposed mechanism, rather than relying solely on correlations between projection geometry and activations.

Thus, the characterization that our optimization interpretation lacks analysis overlooks these existing experiments. We would appreciate clarification of which specific inference the reviewer considers unsupported.

The author’s response fails to explain why "massive activation" serves as a fixed point for the optimizer, and the experiment itself appears potentially susceptible to the fallacy of reverse causality; consequently, my concerns remain unaddressed.

Your comment suggests that you are confused about the primary question our paper investigates. In this paper, we ask whether MAs constitute a fixed point of the optimization (not how that fixed point is reached), i.e., does optimization actively maintain the read–write asymmetry and massive activations once they emerge? Our Hessian and gradient analyses provide strong evidence supporting that the optimizer actively maintains this fixed point, once reached.

Your point that we do not explain "why massive activation serves as a fixed point", concerns their origin. Our paper addresses their persistence. Works such as [1, 2] link MAs to attention sinks, while [3] proposes a mechanism for their emergence. However, we believe these works do not fully explain why training favors such configurations. Why MAs serve as a fixed point remains an open research question, and we welcome concrete experimental suggestions to investigate it.

[1] Sun, M., Chen, X., Kolter, J. Z., & Liu, Z. (2024). Massive activations in large language models. In First Conference on Language Modeling. https://openreview.net/forum?id=F7aAhfitX6

[2] Sun, S., Canziani, A., Lecun, Y. & Zhu, J.. (2026). Anatomy of Massive Activations and Attention Sinks. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:116443-116463 Available from https://proceedings.mlr.press/v306/sun26d.html.

[3] Sun, S., Canziani, A., LeCun, Y., & Zhu, J. (2026). The spike, the sparse and the sink: Anatomy of massive activations and attention sinks. arXiv preprint arXiv:2603.05498.

We hope our response has addressed your concerns and would appreciate your reconsideration of the score in light of these clarifications.

Reviewer_r4xROpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the emergence and persistence of massive activations (MAs) in LLMs by introducing two concepts: read-blindness and write-blindness. Through extensive experiments, the authors identify a systematic read-write asymmetry in MA channels and demonstrate that read-blindness becomes detectable earlier than FFN amplifier specialization during training. Furthermore, the authors show that AdamW updates exert positive pressure on MA magnitude, primarily driven by the accumulated optimizer state. Overall, this work provides complementary insights to Sun et al. (2026) and offers a new perspective on the mechanisms underlying massive activations in Transformers.

[1] Sun S, Canziani A, LeCun Y, et al. The spike, the sparse and the sink: Anatomy of massive activations and attention sinks[J]. arXiv preprint arXiv:2603.05498, 2026.

Critical strengths
  1. This paper introduces a novel perspective based on read-blindness and write-blindness to investigate massive activations in Transformers.
  2. Extensive experiments across multiple language models demonstrate the consistent presence of read-write asymmetry in massive-activation channels.
  3. The finding that read-blindness emerges earlier than FFN amplifier specialization during training provides a novel insight into the formation of massive activations.
  4. The analysis of loss landscape curvature and AdamW optimization dynamics provides further insights into how training may maintain massive activations.
Critical weaknesses
  1. Insufficient causal evidence for the proposed persistence mechanism.

While the experiments demonstrate a systematic read-write asymmetry in massive-activation (MA) channels, the causal relationship between this asymmetry and MA persistence remains insufficiently established. Specifically, read-blindness does not necessarily prevent corrective updates, as a Transformer block can still generate updates to an MA channel using information from other residual-stream coordinates, even without directly reading the MA channel itself. For example, given the residual update hk(l+1)=hk(l)+Fl(h(l))kh_k^{(l+1)}=h_k^{(l)}+F_l\left(h^{(l)}\right)_k, a block that is read-blind to coordinate kk may still produce a corrective update Fl(h(l))k≈−hk(l)F_l\left(h^{(l)}\right)_k \approx-h_k^{(l)} based on information from other coordinates. Therefore, the observed read-write asymmetry alone does not fully explain why massive activations persist across layers. Moreover, the signed-alignment results in Table 2 do not consistently indicate reinforcing updates across models, leaving the claim of continued accumulation insufficiently supported.

  1. Lack of layer-wise analysis of massive-activation dynamics.

Sun et al. (2026) demonstrated that massive activations follow a distinct step-up, plateau, and step-down pattern across Transformer layers: they emerge in early layers, persist through intermediate layers, and are eventually suppressed in the final layers. However, this paper primarily relies on statistics aggregated across layers, which may obscure the different roles of read-blindness and write-blindness at each stage. In particular, the proposed read-blindness mechanism does not adequately explain how massive activations can be suppressed in step-down blocks despite the claimed inhibition of corrective feedback. A layer-wise analysis of read-write asymmetry and its relationship with activation magnitudes would help clarify this issue.

  1. Insufficient evidence that the training objective favors read-blindness.

The Hessian analysis characterizes the local loss geometry of read-side parameters but does not directly establish that restoring read access is penalized by the training objective. I would like to see whether restoring read access systematically increases the loss, and whether this effect is significantly stronger for MA channels than for ordinary channels.

Minor weakness: This article provides a lot of observational information, but I feel it would be better with polishing, especially the abstract.

[1] Sun S, Canziani A, LeCun Y, et al. The spike, the sparse and the sink: Anatomy of massive activations and attention sinks[J]. arXiv preprint arXiv:2603.05498, 2026.

Questions for authors
  1. Layer-wise analysis of read-write asymmetry.

Table 1 reports the median erasure score eraM\text{era}_{\mathcal{M}} along with the rank-biserial correlation rbr_b. If I understand correctly, these statistics are aggregated across all Transformer layers. I would appreciate a more detailed layer-wise analysis, particularly examining how read-write asymmetry evolves across layers and how these changes correlate with the step-up, plateau, and step-down phases of massive activations.

  1. Practical implications of the proposed findings.

This paper provides several novel perspectives on the mechanisms underlying massive activations. However, it remains unclear how these findings can be translated into effective solutions for mitigating massive activations during training. Could the authors further explore or discuss potential approaches to reducing massive activations while preserving model performance?

6 comments

We thank you for your time and efforts in reviewing our work. Please find below our response to your concerns.

Insufficient causal evidence for the proposed persistence mechanism.

Let us clarify. As you pointed out, feedback to a massive-activation (MA) coordinate can follow different pathways. A direct pathway reads the coordinate's own value and uses it to generate a feedback write to that coordinate. In this work, we primarily study the this direct pathway. We posit that training dynamics favor this simple mechanism for producing corrections to the co-ordniates. Given the functional importance of MAs in Transformers [1], suppressing this preferred corrective pathway could help maintain their persistence. As we have shown in our erasure and gradient/Hessian analyses, FFN/attention branches read MA coordinates relatively weakly, despite retaining the ability to write to it. This supports reduction of direct corrective feedback.

Nonetheless, we agree that non-MA coordinates can provide indirect corrective feedback to MA coordinates. Read-blindness weakens the direct pathway while leaving this indirect pathway available. Indeed, our results show that MA coordinates remain largely open to writes from both attention and FFN branches. These writes can depend on other coordinates while being only weakly influenced by the MA coordinates themselves. Indirect corrective feedback is therefore compatible with our read-blind, write-open account.

To test the indirect pathway, we ask whether changes to other coordinates make an attention or FFN branch cancel an MA while the MA's own input value stays fixed? We test intermediate layers (3, 6, 9, 12, 15) of Llama 1.28B. Specifically, we select eight MA targets per layer and branch, giving 80 targets. Each target was paired with a non-MA control at the same token position.

For each target, we change only the other input coordinates at that token. The target coordinate, model weights, other tokens, and residual skip connection remained fixed. Thus, any cancellation had to come from the branch's output. We counted a correction as successful if it reduced the target’s magnitude by at least 90%. We increased the perturbation budget ϵ\epsilon, measured relative to the original token’s RMS over non-MA coordinates. For instance, ϵ=0.1\epsilon=0.1 allows changes with an RMS up to 10% of this reference scale. Each target and its matched control received the same budget.

Perturbation budget ϵ\epsilonMA targets correctedNon-MA controls corrected
00/802/80
0.0010/802/80
0.010/8011/80
0.13/8060/80
0.565/8080/80

Small changes are unable to correct MA co-ordinates. At ϵ=0.1\epsilon=0.1, only 3/80 MA targets were corrected, compared with 60/80 controls. At ϵ=0.5\epsilon=0.5, 65/80 MA targets were corrected, including 27/40 attention targets and 38/40 FFN targets. These results show that indirect correction is possible, but our search generally required substantially larger changes to cancel MAs than controls. Producing such changes would require substantial mechanistic collaboration with the preceding layers, potentially making indirect correction less accessible to optimization. We thus believe model likely disfavors this indirect path.

We also measured language-model loss after applying each perturbation independently. At ϵ=0.5\epsilon=0.5, the mean full-sequence loss increased by 0.0134 for MA targets, compared with 0.00118 for controls. Because the perturbations changed other coordinates, this increase cannot be attributed solely to MA cancellation.

[1] Sun, M., Chen, X., Kolter, J. Z., & Liu, Z. (2024). Massive activations in large language models. In First Conference on Language Modeling. https://openreview.net/forum?id=F7aAhfitX6

Lack of layer-wise analysis of massive-activation dynamics.

Please refer to our response 1 to Reviewer bJh5, where we provide a layerwise analysis of the Llama 1.28B model, reporting median erasure scores, rank-biserial effects (within brackets) against non-massive controls, and maximum activation magnitudes.

Our layerwise analysis reveals a consistent structural pattern. Massive activation coordinates are read less strongly than controls by the value, query, and FFN input projections, while attention and FFN output projections retain comparatively strong write access. This asymmetry generally becomes more pronounced with depth. In the final layers, increasing activation magnitude accompanies higher read-side erasure and lower write-side erasure. For example, between layers 11 and 15, WOW_O erasure decreases from 0.48 to 0.33, while FFN input erasure increases from 0.54 to 0.59.

As we discussed in our response 2 to Reviewer bJh5, the read-blindness is not total, but partial. Consequently, some corrective pathways remain partially open in the intermediate layers of the model. Consequently, intermediate layers of the model can sometimes decrease the massive activations magnitude. However, these decreases do not eliminate massive activations. Large activation values remain throughout the subsequent layers. Our results therefore support a persistent, relative read-blind, write-open asymmetry.

Insufficient evidence that the training objective favors read-blindness.

We respectfully disagree that the Hessian analysis is uninformative about the objective's resistance to restoring read access. Large positive curvature establishes local stiffness, i.e., moving away from this configuration would attract a loss penalty. Our complementary gradient analysis further indicates that optimization favors maintaining massive activation values. Together, these findings support the interpretation that the observed configuration is favored by the training objective. We agree, however, that directly measuring loss after restoring read access provides additional evidence.

We therefore conduct an explicit intervention on the value projections WVW_V. To restore read access across all the features, we replace WV=U\SigmaVTW_V = U\SigmaV^T with its closest orthonormal matrix WV′=UVTW_V'=UV^T via SVD decomposition and scale the matrix to maintain frobenius norm. This intervention reduces the erasure of MA channels, allowing us to measure the accompanying change in batch loss directly.

Batch loss

BeforeAfterDelta
4.225.47+1.25

Loss is weighted by valid next-token target counts; padding is excluded.

W_v input erasure

Cells show the median erasure on M followed by the rank-biserial effect against controls: erasure (rbr_b).

Layer (0-based)Before: erasure (rbr_b)After: erasure (rbr_b)Delta rbr_b
00.53 (+0.29)0.50 (+0.28)-0.01
10.58 (+0.37)0.50 (-0.06)-0.43
20.56 (+0.56)0.50 (-0.07)-0.63
30.55 (+0.55)0.50 (-0.34)-0.89
40.56 (+0.57)0.50 (-0.32)-0.89
50.57 (+0.60)0.50 (-0.47)-1.07
60.60 (+0.63)0.50 (-0.15)-0.78
70.65 (+0.74)0.50 (+0.29)-0.44
80.68 (+0.79)0.50 (-0.40)-1.19
90.66 (+0.84)0.50 (+0.06)-0.79
100.62 (+0.77)0.50 (+0.18)-0.59
110.62 (+0.87)0.50 (-0.19)-1.06
120.62 (+0.95)0.50 (-0.33)-1.27
130.66 (+0.97)0.50 (-0.52)-1.49
140.69 (+0.96)0.50 (-0.72)-1.68
150.68 (+0.92)0.50 (-0.56)-1.48

MA erasure decreased in all 16 layers, while validation loss increased by 1.25. Thus, the intervention successfully reduced measured MA read-blindness and substantially worsened the training objective evaluated on held-out data. This provides experimental evidence, complementary to the local curvature and gradient analyses, that removing the learned attenuation can be costly.

[1] Gu, Xiangming, et al. "When attention sink emerges in language models: An empirical view." International Conference on Learning Representations. Vol. 2025. 2025.

Practical implications of the proposed findings.

Our findings suggest an important direction for mitigating massive activations during training. Our key observation is a read–write asymmetry where both attention and FFN branches are selectively, although partially, read-blind to massive activation coordinates relative to control coordinates, while remaining open to writing into them. This asymmetry provides a mechanism through which massive activations can persist and motivates interventions that prevent selective read-blindness from developing.

Our paper already explores this direction by imposing orthonormal structure on read-side matrices, either through orthonormal initialization followed by freezing or through Cayley-based reparameterization that preserves this structure during training. These interventions affect massive activation magnitudes, providing evidence that modifying the read-side geometry can influence their development. However, the current experiments do not establish a general method for eliminating massive activations while preserving model performance.

A natural next step is to constrain multiple read-side projections together, since intervening on individual projections can leave other pathways available for the asymmetry to develop. Extending Cayley-style constraints to these projections presents computational challenges, particularly for the large rectangular FFN matrices. Developing efficient structured or approximate constraints, and evaluating their trade-offs between activation magnitude, model performance, and training cost, is therefore a promising direction for future work.

Reviewer_bJh5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes how large activations in residual stream directions appear and persist throughout an LM. The authors introduce metrics to measure the "read/write-blindness" of attention and MLP blocks of the network, which give an indication of how much a block reads from and writes to the direction in the residual stream carrying the large activation. They also analyse the Hessian of the loss to see whether directions that read from/write to massive activations have special properties.

Critical strengths

Analyzing read and write blindness of particular directions in the residual stream, including the massive activations, is a promising direction of inquiry. As is the Hessian w.r.t the directions related to these directions. I'm unaware of past work using these techniques to give an account of massive activations.

Critical weaknesses
  1. The erasure metric which underpins the paper is stated to be a max over all layers. This means that a high erasure score may be a result of only a single read-blind layer. This metric does not validate claims that MAs are “systematically ignored” across layers, or that read-blindness explains their persistence.

  2. The numbers presented in table 1 state that an erasure of WV, WQ, Wgate/Wup are only 0.75, 0.65, and 0.63, respectively, for the Llama 1.28B (Table 1). This does not establish that MAs are functionally ignored.

  3. In section 4.2 a model is pretrained with fixed W_{gate} and W_{up} matrices. The authors report that this actually improves the perplexity compared to the baseline pretrain by 0.54. Improving perplexity while fixing half of each block’s parameters is an extremely unusual result, making me suspicious that the results in this section — and the rest of the paper which also uses the custom trained model — may be specific to models which may not have been well-trained.

Questions for authors

None

3 comments

Please find our response to your concerns below.

The erasure metric which underpins the paper is stated to be a max over all layers. This means that a high erasure score may be a result of only a single read-blind layer. This metric does not validate claims that MAs are “systematically ignored” across layers, or that read-blindness explains their persistence.

Thank you for raising this concern. In the paper, we report max over layers as an aggregate statistics for erasure to establish a model level read-write asymmetry. Below, we provide a layer wise analysis. Specifically, we report median Erasure scores for massive co-ordinates along with their rank-biserial (within brackets) over control co-ordinates. We report these metrics for different attention and FFN matrices at each layer in Llama 1.28B model. The table is given below.

Across layers, massive coordinates exhibit increasing read/write asymmetry: value, query, and FFN input projections increasingly underweigh these coordinates relative to controls, whereas attention and FFN output projections preferentially target them. Activation maxima emerge early, decline through intermediate layers, and rise sharply near the output. These results support selective read-side blindness alongside an open write-side connectivity.

LayerValue projection WVW_VAttention output WOW_OQuery projection WQW_QKey projection WKW_KFFN gate/up WgateW_\text{gate}, WupW_\text{up}FFN down projection WdownW_\text{down}Max magnitude
00.53 (+0.29)0.49 (-0.64)0.51 (+0.21)0.52 (+0.38)0.52 (+0.35)0.48 (-0.86)152
10.58 (+0.37)0.48 (-0.86)0.55 (+0.61)0.48 (-0.40)0.51 (+0.53)0.48 (-0.74)203
20.56 (+0.56)0.48 (-0.71)0.52 (+0.71)0.49 (-0.28)0.51 (+0.63)0.46 (-0.97)845
30.55 (+0.55)0.48 (-0.67)0.51 (+0.44)0.51 (+0.11)0.51 (+0.66)0.47 (-0.81)837
40.56 (+0.57)0.47 (-0.84)0.53 (+0.59)0.50 (-0.05)0.52 (+0.69)0.47 (-0.95)840
50.57 (+0.60)0.47 (-0.85)0.53 (+0.59)0.50 (-0.14)0.53 (+0.75)0.47 (-0.99)835
60.60 (+0.63)0.46 (-0.64)0.55 (+0.81)0.51 (+0.06)0.54 (+0.82)0.47 (-1.00)778
70.65 (+0.74)0.47 (-0.47)0.56 (+0.81)0.50 (-0.15)0.54 (+0.86)0.46 (-1.00)733
80.68 (+0.79)0.47 (-0.40)0.55 (+0.92)0.50 (-0.10)0.54 (+0.90)0.46 (-0.97)672
90.66 (+0.84)0.48 (-0.47)0.54 (+0.82)0.50 (-0.17)0.53 (+0.88)0.46 (-0.95)607
100.62 (+0.77)0.48 (-0.36)0.53 (+0.93)0.51 (+0.07)0.53 (+0.90)0.45 (-0.94)521
110.62 (+0.87)0.48 (-0.45)0.54 (+0.90)0.49 (-0.31)0.54 (+0.93)0.45 (-0.96)436
120.62 (+0.95)0.48 (-0.44)0.54 (+0.83)0.49 (-0.30)0.54 (+0.93)0.44 (-0.91)485
130.66 (+0.97)0.44 (-0.58)0.55 (+0.88)0.48 (-0.35)0.56 (+0.96)0.43 (-0.83)574
140.69 (+0.96)0.36 (-0.65)0.56 (+0.95)0.48 (-0.31)0.58 (+0.96)0.44 (-0.65)746
150.68 (+0.92)0.33 (-0.86)0.58 (+0.85)0.49 (-0.02)0.59 (+0.96)0.41 (-0.76)2266

Massive activations emerge early, with maximum magnitude reaching 845 at layer 2. Maxima remain between 436 and 845 through layers 2–14 before rising sharply to 2266 in the final layer. Our model therefore exhibits early emergence and persistence, but differs from the final-layer step-down reported by Sun et al [1]. Note that this behavior is not unique for our models as we observe a similar pattern in Olmo3 8B [2].

[1] Sun, Shangwen, et al. "The spike, the sparse and the sink: Anatomy of massive activations and attention sinks." arXiv preprint arXiv:2603.05498 (2026). [2] Olmo, Team, et al. "Olmo 3." arXiv preprint arXiv:2512.13961 (2025).

The numbers presented in table 1 state that an erasure of WV, WQ, Wgate/Wup are only 0.75, 0.65, and 0.63, respectively, for the Llama 1.28B (Table 1). This does not establish that MAs are functionally ignored.

Our use of “read-blindness” refers to massive coordinates being read less strongly than other coordinates, rather than being completely ignored. An erasure score of 1 indicates complete blindness, whereas intermediate scores measure how weakly a coordinate is read relative to the matrix's average response across coordinates.

For this reason, Table 1 reports both the median erasure and the rank-biserial effect against non-massive controls. For Llama 1.28B, these effects are +0.99, +0.99, and +1.00 for WVW_V, WQW_Q, and the FFN gate/up projections, respectively. These effects show that massive coordinates consistently have higher erasure than controls. Thus, our claim concerns their distinctive treatment by the learned read projections, rather than whether their erasure scores approach 1.

Partial reading is part of our explanation. As discussed in Section 5, the FFN retains some read access to massive coordinates and can still produce strong amplification. This is consistent with our central observation: massive coordinates are read less strongly than other coordinates, while their write pathways remain open. We will clarify this relative meaning of “read-blindness” and revise wording accordingly.

In section 4.2 a model is pretrained with fixed W_{gate} and W_{up} matrices. The authors report that this actually improves the perplexity compared to the baseline pretrain by 0.54. Improving perplexity while fixing half of each block’s parameters is an extremely unusual result, making me suspicious that the results in this section — and the rest of the paper which also uses the custom trained model — may be specific to models which may not have been well-trained.

Our experimental setup closely follows [1]. All models are trained on 5B tokens over 20,000 updates. In the paper, we report perplexity at the final checkpoint since training loss continued to decrease throughout training. However, decreasing training loss does not ensure that the final checkpoint achieves the best validation performance. To assess whether the result depends on checkpoint selection, we instead select the checkpoint with the lowest validation loss for each model, applying the same criterion across all runs. The results are as follows.

ModelPerplexityChange from baseline
Llama 1.28B11.91—
WVW_V frozen12.37+0.46
WVW_V reparameterized11.97+0.06
Wup,WgateW_{\mathrm{up}}, W_{\mathrm{gate}} frozen12.47+0.56

As expected, all three interventions yield higher perplexity than the baseline. In particular, freezing WupW_{\mathrm{up}} and WgateW_{\mathrm{gate}} increases perplexity by 0.56, reversing the improvement you pointed out. The smallest degradation occurs with WvW_v reparameterization, which allows updates while preserving the orthonormal constraint. Both frozen variants incur larger degradation, consistent with their reduced trainable capacity.

We pledge to update Probing Experiments (Section 4.2) and implementation details (Appendix E) sections of the paper with these results.

[1] Gu, Xiangming, et al. "When attention sink emerges in language models: An empirical view." International Conference on Learning Representations. Vol. 2025. 2025.

I thank the authors for engaging with my critiques. Their subsequent results would improve the paper a little, though not enough for me to change my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.