acceptodds
Under review as a conference paper at ICLR 2027

Magnitude-Induced Rank Collapse in FFN Value-Vector Gradients

Abstract

Each column of the FFN down-projection maps an intermediate channel to the residual stream in a Transformer model. How individual training sequences contribute to updating these columns is obscured by minibatch aggregation and model- or layer-level gradient summaries. We therefore characterize each column's per-sequence gradient matrix using normalized energy effective rank (NEER), which responds to both directional correlation and gradient-norm imbalance. We first observe that columns within the same layer exhibit a pronounced low-rank tail across optimization, showing that sequence contributions can differ sharply across parameter vectors. We further find that low NEER often reflects gradient-magnitude imbalance rather than directional alignment, with many low-NEER columns retaining substantial directional diversity. To trace the source of these outliers, we use the standard activation–backpropagation factorization. It reveals that large column-specific activations and strong residual-space backpropagated signals coincide at the same token positions, whereas neither factor alone explains the concentration. Together with a simplified optimization analysis, these findings suggest that high-magnitude sequence gradients in some columns can suppress otherwise diverse directions and limit their contribution to optimization. Motivated by this analysis, we test parameter-local selective clipping of dominant sequence gradients in low-NEER columns. Across Llama-3.1-8B, Llama-3.2-3B, and Mistral-7B, selectively rebalancing dominant local contributions yields consistent downstream gains, suggesting that reducing parameter-local gradient imbalance can be beneficial during fine-tuning.

Primary area:
optimization
Keywords:
Transformer Feed-Forward Networks, Per-Example Gradients, Effective Rank, Parameter-Sample Interaction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EhT9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how individual training sequences contribute to gradients of individual columns of Transformer FFN down-projection matrices. For each column, it forms a matrix of per-sequence gradients and summarizes its spectrum using normalized energy effective rank (NEER). It reports substantial within-layer NEER heterogeneity and persistent low-NEER tails across training stages and models.

The paper derives an exact decomposition of the NEER deficit into sequence-energy concentration and additional directional correlation. Norm equalization leaves gradient directions intact and substantially restores NEER for many low-NEER columns, suggesting that magnitude imbalance, rather than directional collapse alone, explains much of the observed low rank. A token-level activation/backpropagation factorization and proxy comparisons indicate that the observed pairing of column activations and residual-space backpropagated signals helps explain which sequences dominate local gradient energy. Finally, the authors selectively clip the largest sequence-gradient contribution in low-NEER columns. On three models, this yields modest improvements in average accuracy over four mathematical reasoning benchmarks, with a mixed per-benchmark pattern. The main contribution is the gradient-geometry diagnosis; the intervention provides initial, but not conclusive, evidence that rebalancing can be useful.

Critical strengths
  1. A clear and useful decomposition of low effective rank. The exact split into energy concentration and directional correlation, together with norm equalization that preserves directions, prevents a common interpretive error: treating every low-rank gradient matrix as evidence that examples agree on direction. This is the paper’s most consequential contribution.
  2. A coherent, multi-stage diagnostic study. The authors connect the local geometry to token-level activation/backpropagation pairing and test the distributional pattern across models, stages, datasets, and independently sampled minibatches. This gives the descriptive claim more support than a single-checkpoint case study.
Critical weaknesses
  1. The optimization experiment does not isolate the contribution of NEER-based selection. The paper compares selective local clipping with a baseline, but not with standard global/parameter-wise clipping or a control that clips the same number of sequence contributions in randomly selected columns. Thus, the experiment supports the plausibility of local rebalancing, while leaving open whether NEER-based targeting is necessary for the observed gains. This limits the intervention claim, not the paper’s principal diagnostic contribution.
  2. The scope and operating cost of the intervention remain narrow. The downstream intervention is tested on mathematical fine-tuning and four math benchmarks, with mixed per-benchmark outcomes for Mistral-7B; the method also needs explicit per-sequence gradients and local NEER computation, whose overhead is not quantified. The authors appropriately frame it as a mechanism test and note the need for efficient surrogates, but the current evidence does not establish a practical optimizer recipe.
Questions for authors

Clarification requests

  1. Did you evaluate a matched standard-clipping control (global norm clipping, per-parameter clipping, or clipping the same number of top sequence contributions in randomly selected columns)? Such a control would clarify whether NEER-based local selection is responsible for the improvement beyond generic clipping.
  2. What is the additional wall-clock time and peak memory for computing per-sequence, per-column gradients and applying the intervention, relative to the baseline training setup?
  3. How sensitive are the downstream results to the intervention settings (the selected column fraction and number of clipped sequences per column), given that the paper uses ρ=5%\rho=5\% and k=1k=1 by default?

Suggestions for future work

  1. A lower-cost online proxy for identifying local magnitude dominance, evaluated against the explicit per-sequence-gradient method, would help determine whether the proposed mechanism can translate into a practical training algorithm.
Reviewer_UsJGOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Authors propose a metric, normalized energy effective rank a.k.a. NEER, to study "Magnitude-Induced Rank Collapse in FFN Value-Vector Gradients". Authors evaluate this metric on LLaMA and Mistral models. Experient results only show marginal improvements with selective local clipping.

Critical strengths

Design is simple and straightforward, which is good. And experiments result matches what it is expected to be like.

Critical weaknesses
  1. Motivation is unclear. I get it that authors tried to understand "how different sequences contribute to the same local parameter vector—and how this structure varies across vectors within a layer", but the importance of the research question itself remains unclear. Authors should well present their motivation and demonstrate the research questions are not made up.

  2. Paper is hard to follow. This paper is poorly-written, especially Introduction. In this paper, there are lots of made-up terms can confuse authors and authors (if they were aware) should avoid using those slops. One example is Singular Value Energy Spectrum (line 052). Besides, I found authors used the word energy multiple times without a clear definition to demonstrate the concept.

  3. Biased foundation models and insignificant performance improvement. In this paper, authors aim to study "Magnitude-Induced Rank Collapse in FFN Value-Vector Gradients", and authors choose Mistral and LLaMA as the foundation models for experiments. However, FFNs of Mistral and LLaMA is highly similar, in which both using up/gate/down projection matrices and SwiGLU as activation. Therefore, conclusion derived from authors' experiments are not sound to some extents. Additionally, performance improvement reported in Table 3 is insignificant, and the improvement looks more like sorts of turbulence.

Questions for authors

No questions

2 comments
AuthorsOpenReview ↗

Clarification of the motivation and the term “energy”

R2-W2 “Paper is hard to follow. This paper is poorly-written, especially Introduction. In this paper, there are lots of made-up terms can confuse authors.[...] One example is Singular Value Energy Spectrum (line 052). Besides, I found authors used the word energy multiple times without a clear definition to demonstrate the concept.”

We will revise the introduction to explain our research motivation and the choice of NEER more clearly, as outlined in our response to all reviewers above. We will also define terminology before using it. In particular, the term energy means the squared Euclidean norm of a vector in this paper. This terminology follows the established squared-norm convention in signal processing and related mathematical treatments, and gives us a concise way to discuss both sample-level and spectral contributions. We will state this definition explicitly at its first use.

AuthorsOpenReview ↗

Low-NEER patterns also occur in non-SwiGLU pretraining

R2-W3 “Biased foundation models and insignificant performance improvement. [...] However, FFNs of Mistral and LLaMA is highly similar, in which both using up/gate/down projection matrices and SwiGLU as activation. Therefore, conclusion derived from authors' experiments are not sound to some extents.”

We ran additional experiments on a GPT-2-style GELU nanoGPT model with approximately 124M parameters, trained on OpenWebText. We used B=48B=48, sequence length 512, and data seeds 401/402/403. At layers 0, 5, and 11, we selected the bottom 5% of WdownW_{\mathrm{down}} columns by raw NEER and compared the same vectors before and after norm equalization. The results are shown below.

StepLayerAll: rawBottom 5%: rawSame vectors: unit-norm
50000.5449 ± 0.03670.3560 ± 0.12340.6421 ± 0.0197
50050.6086 ± 0.09180.3144 ± 0.18710.8188 ± 0.0065
500110.6236 ± 0.02990.2981 ± 0.06250.8453 ± 0.0026
100000.7314 ± 0.03710.5232 ± 0.15950.8591 ± 0.0023
100050.7162 ± 0.01990.5194 ± 0.07890.8310 ± 0.0091
1000110.6748 ± 0.00580.4430 ± 0.02350.8259 ± 0.0077
500000.8164 ± 0.00940.5355 ± 0.03360.9426 ± 0.0019
500050.5428 ± 0.01800.2196 ± 0.02830.7958 ± 0.0140
5000110.5384 ± 0.01420.1792 ± 0.01180.8405 ± 0.0089

The table shows that low NEER remains evident in a non-SwiGLU model, with NEER increasing after norm equalization across the examined checkpoints and layers. Together with the Llama and Mistral results, this supports the conclusion that the observed phenomenon extends beyond a single model family or SwiGLU architecture.

Reviewer_JXyFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at the per sequence impact on the gradients of the MLP weights in an LLM. The analysis, however, is focused on each weight column separately. First, they define a function that measures the rank of the gradient received by each column from each sequence. Then, they point to some columns having extremely low-rank gradients, indicating that they're updated by a small subset of the sequences. Then, they show that this is due to the gradient received by those columns being very large for some sequences, resulting in a significant rank deficiency. They investigate the mechanism through which this occurs, and later propose a solution for it that improves fine-tuning on a single dataset and for three, medium-scale models.

Critical strengths
  • The analysis is interesting, and to the best of my knowledge somewhat novel.
  • The problem seems to be significant. I believe the issue of LLMs learning low-rank weights is well-known. However, the paper doesn't really connect the observe phenomenon, which is gradient related as opposed to weight related, to other similar issues.
Critical weaknesses
  • The paper is not easy to read. Some notation and abbreviations are not explained before being used extensively by the paper. The same is true about some functions and measures the paper investigates. As an example, the total variation measure appears out of nowhere in lines 401-405. There's no mathematical definition for it and no intuitive explanation for what it represents.
  • The way the authors define NEER, which is the exponential of the entropy of the normalized squared singular values, is technically not equivalent to effective rank. And the paper does not really motivate the choices of functions in this definition.
  • The way NEER and the other measures are presented for the layers is less informative than it could be, and I think the authors need to improve the presentation a bit. The tables only report certain statistics (like the bottom 1% median, median, ratio of the two in Table 1, or the low-NEER group alone in Table 2). No absolute reference is given for NEER and what should be considered a collapsed value. The authors also don't really motivate what would be an ideal or desirable NEER value for the gradients.
  • I think the paper doesn't do a good job to convince the reader about the utility of looking into the tail of the NEER distribution. The only evidence for low-NEER columns being important is their gradient energy visible to the optimizer, which appears near the end of the paper. The authors should restructure the paper so that it would be used as a motivation for the analysis that follows. Furthermore, the differences seem very modest.
  • Unless I'm missing something, most of the results in the paper are done on a single batch (lines 215-225). Therefore, I'm wary of agreeing with any of the findings in the paper unless the authors increase the sample size for these experiments and report the variation.
  • The paper suffers a bit from over-formalizing some things. For example, the equations and the measures in equation (10) and lines 290-291 are completely unnecessary in my opinion. The same can be said about equation (8).
  • The entirety of section 4 is dedicated to one thing: proving in three different ways that magnitude differences is the main reason behind the low NEER problem. I believe one analysis would've sufficed to do so. We don't need one table and three panels to prove the same thing. Some of this content belongs to the appendix.
  • The paper is overloading the term rank-collapse, which is an important term related to signal propagation problems in deeper neural networks. I think that makes the title and the content a bit confusing.
  • The analysis in section 5 doesn't really earn the title of the section. The authors were able to show that for the sequences contributing the most to the low-NEER columns, there's a coupling between the magnitude of the activations and the gradient with respect to the tokens for the tokens that contribute the most to the gradient w.r.t. W_down provided by that sequence. This only explains the mechanism through which the low-NEER phenomenon happens. Not why.
  • The main metric for the NEER analysis being useful is the corresponding intervention that mitigates the problem with low-NEER columns improving the performance of the model. The authors provide some fine-tuning results for a single dataset on 3 different models. This experiment is not convincing at all. Firstly, I would argue that for such a problem, a better benchmark would be pre-training results on smaller models. Furthermore, one dataset is simply not enough to make any conclusions. And there's no analysis that follows the results. Namely, where does the improvement come from? For example, as an speculation on my part, it could be that the model is spreading the updates over parameters more efficiently. But that would indicate forgetting.
Questions for authors
  • Can the authors explain where NEER comes from? And what the motivation behind this formulation is? I found a similar function in [1] (algorithm 2), but it's not explained there either.
  • Can the authors explain, what would we gain from investigating the entity defined in equation (8) and also proposition 1 that we wouldn't otherwise gain from investigating the entity in equation (9)? If we aim to distinguish the effect of magnitude from direction, then (9) would suffice because it neutralizes the effect of magnitude, no?

References

[1] Yan, S., Li, Z., Zhou, C., Huang, Q., Liu, K. and Zhang, M., 2026. Toward Better Geometric Representations for Molecule Generative Models.

4 comments
AuthorsOpenReview ↗

Clarification of notation

R1-W1 “Some notation and abbreviations are not explained before being used extensively by the paper. The same is true about some functions and measures the paper investigates. As an example, the total variation measure appears out of nowhere in lines 401-405. There's no mathematical definition for it and no intuitive explanation for what it represents.”

In the revision, we will explain the notation and abbreviations more clearly before using them. For example, the total variation (TV) distance used to compare exact and proxy sequence-energy distributions is

=\frac12\sum_s\left|q_s^{\mathrm{Exact}}-q_s^{\mathrm{proxy}}\right|.$$ TV lies in $[0,1]$, with smaller values indicating a closer match to the exact energy allocation. A value of 0 means that the distributions are identical, while 1 means that their nonzero supports are disjoint. It can also be interpreted as the minimum probability mass that must be reassigned to turn one distribution into the other. For example, the TV distance between $(0.8,0.2)$ and $(0.5,0.5)$ is $0.3$.
AuthorsOpenReview ↗

Squared singular values enable a sample-energy decomposition

R1-W2 “The way the authors define NEER, which is the exponential of the entropy of the normalized squared singular values, is technically not equivalent to effective rank. And the paper does not really motivate the choices of functions in this definition.”

We use normalized squared singular values rather than normalized singular values for two reasons.

First, squared singular values correspond to additive energy. The identity below links total sample-gradient energy to its decomposition across orthogonal spectral directions. Thus, pjp_j is the fraction of energy carried by spectral direction jj. It is also a normalized eigenvalue of the uncentered second-moment matrix XX⊤/BXX^\top/B.

∣X∣F2=∑s∣gs∣22=∑jσj(X)2.\\|X\\|_F^2=\sum_s\\|g_s\\|_2^2=\sum_j\sigma_j(X)^2.

Second, the energy-based definition supports a direct sample-level decomposition. Using the sample-energy distribution qq and spectral-energy distribution pp defined above, we obtain

=\underbrace{\log B-H(q)}_{C^E} +\underbrace{H(q)-H(p)}_{C^D}.$$ Here, $C^E$ measures unequal energy allocation across samples, and $C^D$ measures the additional spectral concentration associated with their directional relationships under those energy weights. This directly addresses our question: does low effective rank arise because a few samples have much larger energies, or because their gradient directions are concentrated? The unsquared definition also responds to magnitude and direction, but does not yield this decomposition directly in terms of sample squared norms.
AuthorsOpenReview ↗

Low-NEER patterns recur across batches and pretraining checkpoints

R1-W5 “Unless I'm missing something, most of the results in the paper are done on a single batch (lines 215-225). Therefore, I'm wary of agreeing with any of the findings in the paper unless the authors increase the sample size for these experiments and report the variation.”

Several main-text comparisons use the same batch to isolate the effects of model state or diagnostic transformations. The submitted appendix already includes experiments that vary batch composition and batch size, as well as results on mathematics and code data. Specifically, Appendix B.2, “Robustness across random minibatches” (p. 14, lines 730–746; Table 8 and Figure 5) repeats the analysis on independently sampled MetaMathQA batches at fixed B=48B=48. Appendix B.4, “Effect of the number of sampled sequences” (p. 16, lines 850–856; Table 10 and Figure 6) varies BB over 16,32,48,64,8016,32,48,64,80. Appendix B.5, “Results on an additional data domain” (p. 16, lines 860–863; Table 11 and Figure 7) reports the CodeFeedback results. Tables 8, 10, and 11 are on p. 23 of the submitted PDF.

We additionally ran experiments on a GPT-2-style GELU nanoGPT model with approximately 124M parameters, trained on OpenWebText. We used B=48B=48, sequence length 512, and data seeds 401/402/403. At layers 0, 5, and 11 of each checkpoint, we selected the bottom 5% of WdownW_{\mathrm{down}} columns by raw NEER and compared the original gradients with their norm-equalized counterparts.

StepLayerAll: rawBottom 5%: rawSame vectors: unit-norm
50000.5449 ± 0.03670.3560 ± 0.12340.6421 ± 0.0197
50050.6086 ± 0.09180.3144 ± 0.18710.8188 ± 0.0065
500110.6236 ± 0.02990.2981 ± 0.06250.8453 ± 0.0026
100000.7314 ± 0.03710.5232 ± 0.15950.8591 ± 0.0023
100050.7162 ± 0.01990.5194 ± 0.07890.8310 ± 0.0091
1000110.6748 ± 0.00580.4430 ± 0.02350.8259 ± 0.0077
500000.8164 ± 0.00940.5355 ± 0.03360.9426 ± 0.0019
500050.5428 ± 0.01800.2196 ± 0.02830.7958 ± 0.0140
5000110.5384 ± 0.01420.1792 ± 0.01180.8405 ± 0.0089

The phenomenon remains present in these experiments. Together with the batch and domain analyses in the appendix, these results support a recurring low-NEER pattern across the examined optimization settings, rather than an isolated batch-specific observation.

AuthorsOpenReview ↗

Eq. (8) retains energy-allocation information removed by Eq. (9)

R1-W6 “The paper suffers a bit from over-formalizing some things. For example, the equations and the measures in equation (10) and lines 290-291 are completely unnecessary in my opinion. The same can be said about equation (8).”

R1-W7 “The entirety of section 4 is dedicated to one thing: proving in three different ways that magnitude differences is the main reason behind the low NEER problem. I believe one analysis would've sufficed to do so. We don't need one table and three panels to prove the same thing. Some of this content belongs to the appendix.”

R1-Q2 “Can the authors explain, what would we gain from investigating the entity defined in equation (8) and also proposition 1 that we wouldn't otherwise gain from investigating the entity in equation (9)? If we aim to distinguish the effect of magnitude from direction, then (9) would suffice because it neutralizes the effect of magnitude, no?”

Regarding Section 4, Figures 3(a) and 3(b) (p. 7, caption at lines 335–342; discussed in Section 4.2, p. 6, lines 296–304) present similar information, so we will shorten this part in the revision. However, Eq. (8) does provide information that Eq. (9) cannot. Besides identifying the cause of low NEER, Section 4 also examines how energy is distributed across samples. Equation (9) can show that magnitude differences are the main cause of low NEER, but what does the original gradient-energy distribution look like? Equation (9) discards this information through normalization. An intuitive way to understand Eq. (8) is to consider an idealized sample-gradient matrix with equal norms and mutually orthogonal directions, whose NEER is 1. Suppose the actual NEER is 0.2, indicating a less uniform distribution than in the ideal case. The terms CEC^E and CDC^D quantify this nonuniformity in terms of energy and angles. A larger CE/(CE+CD)C^E/(C^E+C^D) means that, under the current energy allocation, the maximum attainable NEER is smaller even if the directions are made mutually orthogonal. This depends on the specific energy allocation and cannot be captured by Eq. (9) alone. Similarly, Section 4.3 examines energy concentration directly, making it easier to see how a small number of samples dominate the total energy.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to all reviewers

We thank the reviewers for their time and thoughtful feedback. Your comments have helped us identify where the motivation, terminology, and presentation of our findings need to be clearer. We first clarify the motivation and our choice of metric, then address the specific concerns below.

Clarification of the research motivation

We would first like to explain our research motivation in more detail. Our study examines gradient contributions along two axes: (1) contributions from different samples, and (2) contributions to different parameter vectors, primarily the columns of WdownW_{\mathrm{down}} in the submitted paper.

(1) Gradient contributions from different samples. When combining observations, we typically examine not just their mean but also properties such as their variance and heterogeneity. This is a well-established perspective in statistics and related fields. A similar question arises in optimization. Different samples, such as different mathematical problems, undergo their respective forward passes and produce their own gradients. Standard training aggregates these gradients according to the weighting prescribed by the loss. We seek to go one step further by examining the internal structure of this aggregation and exploring improvements motivated by what we find (equivalently, this can be viewed as asking how to assign weights to different samples more effectively, rather than always weighting them equally).

(2) Gradient contributions to different parameter vectors. Our choice of analysis unit follows the computation performed by an LLM. A matrix–vector product can be written as a linear combination of matrix columns, or of rows under the transposed convention. The interpretation of FFNs as key-value memories provides a mechanistic basis for this perspective. During backpropagation, each such unit, a parameter vector associated with a channel, receives a well-defined gradient that describes the update signal acting on its weights. Geva et al., EMNLP 2021.

NEER jointly captures gradient magnitude and direction

We next explain why we use normalized energy effective rank (NEER).

We wish to characterize a collection of high-dimensional gradient vectors. Magnitude and direction are two natural aspects of this characterization. A gradient's norm describes the magnitude of a sample's contribution to the update signal for a parameter vector. In our paper, energy means the squared Euclidean norm, following standard terminology in signal analysis; we will make this definition explicit in the revision. Pairwise angles describe the geometric relationships between the update signals contributed by different samples. With fixed parameter coordinates and the Euclidean metric, a nonzero sample gradient can be written as follows.

gs=rsus,rs=∥gs∥2,∥us∥2=1.g_s=r_su_s,\qquad r_s=\|g_s\|_2,\qquad \|u_s\|_2=1.

The squared norm of the aggregate makes the roles of magnitude and direction explicit: the first term sums individual gradient energies, while the second captures reinforcement or cancellation between sample contributions.

= \sum_{s} r_{s}^{2} + 2\sum_{s \lt t} r_{s}r_{t}\langle u_{s},u_{t}\rangle.$$ Spectral measures naturally combine these two aspects, which motivates our choice of NEER. Let $X=[g_1,\ldots,g_B]\in\mathbb{R}^{d\times B}$ and $K=X^\top X$. The diagonal entries of this Gram matrix are gradient energies, while its off-diagonal entries encode direction similarities weighted by the gradient norms. Under the paper's setting $B\le d$ and $\|X\|_F>0$, define $\lambda_j=\sigma_j(X)^2$, including zero eigenvalues. The energy proportions $p_j$ and NEER are $$p_j=\frac{\lambda_j}{\sum_k\lambda_k},\qquad \operatorname{NEER}(X)=\frac{\exp\left(-\sum_jp_j\log p_j\right)}{B}, \qquad 0\log0:=0.$$ Thus, $B\cdot\operatorname{NEER}(X)$ describes the effective number of spectral directions over which this collection of sample gradients distributes its energy. NEER expresses that number relative to the batch size. We would also like to clarify that the spectral-entropy formulation underlying NEER is established in the literature. Related effective-rank and diversity measures have been used in several settings, including analyses of sample diversity. **(1) PCA.** Principal component analysis also uses a matrix spectrum to describe how information is distributed across directions. Its principal axes are eigenvectors of the covariance matrix, and its explained-variance ratios have the same normalized squared-singular-value form as $p_j$. Standard PCA centers the observations first. We retain uncentered gradients because our analysis concerns the contributions that form the actual aggregate gradient supplied to the optimizer. [Jolliffe and Cadima, 2016, §2](https://pmc.ncbi.nlm.nih.gov/articles/PMC4792409/). **(2) Vendi Score.** Vendi Score measures the diversity of a collection through the spectrum of a similarity matrix. NEER has a direct relationship to its probability-weighted version: use the cosine Gram matrix of gradient directions as the similarity kernel, weight samples by their gradient-energy shares, and divide the resulting Vendi Score by $B$. Specifically, for $C_{st}=\langle u_s,u_t\rangle$, $q_s=\|g_s\|_2^2/\sum_r\|g_r\|_2^2$, and $D_q=\operatorname{diag}(q)$, $$D_q^{1/2}CD_q^{1/2}=\frac{K}{\operatorname{tr}K}, \qquad \operatorname{NEER}(X)=\frac{\operatorname{VS}(C,q)}{B}.$$ This correspondence follows directly from the definitions. [Friedman and Dieng, 2023, Appendix 7.1, Definition 7.1](https://arxiv.org/html/2210.02410#S7.SS1). **(3) G-Vendi / Prismatic Synthesis.** This work uses gradient NEER after norm normalization to study LLM data diversity and generalization. We additionally account for gradient magnitudes, which is the focus of our paper. [Jung et al., *Prismatic Synthesis*, §2.1](https://arxiv.org/html/2505.20161v1#S2.SS1). **(4) Learning Continually by Spectral Regularization.** This work also uses the effective rank of a matrix of per-example gradients to characterize gradient diversity. Its spectral weights are normalized singular values, $\sigma_j/\sum_k\sigma_k$, rather than squared singular values. We explain our choice of the squared version in [R1-W2] below. [Lewandowski et al., §3.3](https://arxiv.org/html/2406.06811v2#S3.SS3). We will explain this motivation for choosing NEER more clearly in the revision.

Additional evidence: Low-NEER patterns also occur in other FFN and attention components

We have also extended the NEER analysis beyond WdownW_{\mathrm{down}} vectors to other LLM components, with layer 13 shown below. Results for layers 0 and 27 will be included in the revision.

LayerComponentRaw NEERUnit-norm NEERRelative increase (%)
13down0.3544 ± 0.01470.9397 ± 0.0239165.1%
13up0.2824 ± 0.01480.7855 ± 0.0408178.1%
13gate0.1851 ± 0.01900.7592 ± 0.0463310.2%
13Q0.2757 ± 0.02580.7629 ± 0.0116176.8%
13K0.3336 ± 0.02580.7468 ± 0.0503123.9%
13V0.2756 ± 0.02780.5821 ± 0.0342111.2%
13O0.3946 ± 0.04670.9629 ± 0.0075144.0%

Raw NEER uses the original per-sample gradients of each parameter vector; Unit-norm NEER normalizes each of these gradients to unit length, preserving its direction, before computing NEER. Both are averaged over the same bottom 5% of parameter vectors selected by Raw NEER within each component. Relative increase is (Unit-norm NEER/Raw NEER−1)×100%(\text{Unit-norm NEER}/\text{Raw NEER}-1)\times100\%, using the unrounded means.

The phenomenon also appears across other components, with clear differences between them. For example, the Q/K/V projections in layer 0 exhibit smaller absolute increases in NEER after norm equalization than the FFN components, suggesting a different balance between magnitude and directional effects. Understanding these component-specific differences is a direction for our future work.

Clarification of the optimization implications

Taken together, we view the core contribution as a more detailed analysis of information that can be obscured by standard gradient aggregation, together with a parameter-local account of how samples contribute to different parts of the model. We expect that more effective optimization strategies may be developed from the low-NEER phenomenon. The intervention presented in the paper is preliminary, and its experimental noise requires better control. Our future work will improve the experimental setup and investigate optimization more systematically, including isolating the effects of optimizer momentum and second-moment states when studying clipping, and selecting vectors according to the predicted spectral effect of clipping rather than their current NEER alone. We also consider the theoretical and interpretive findings valuable subjects of study in their own right.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.