- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how individual training sequences contribute to gradients of individual columns of Transformer FFN down-projection matrices. For each column, it forms a matrix of per-sequence gradients and summarizes its spectrum using normalized energy effective rank (NEER). It reports substantial within-layer NEER heterogeneity and persistent low-NEER tails across training stages and models.
The paper derives an exact decomposition of the NEER deficit into sequence-energy concentration and additional directional correlation. Norm equalization leaves gradient directions intact and substantially restores NEER for many low-NEER columns, suggesting that magnitude imbalance, rather than directional collapse alone, explains much of the observed low rank. A token-level activation/backpropagation factorization and proxy comparisons indicate that the observed pairing of column activations and residual-space backpropagated signals helps explain which sequences dominate local gradient energy. Finally, the authors selectively clip the largest sequence-gradient contribution in low-NEER columns. On three models, this yields modest improvements in average accuracy over four mathematical reasoning benchmarks, with a mixed per-benchmark pattern. The main contribution is the gradient-geometry diagnosis; the intervention provides initial, but not conclusive, evidence that rebalancing can be useful.
Critical strengths
- A clear and useful decomposition of low effective rank. The exact split into energy concentration and directional correlation, together with norm equalization that preserves directions, prevents a common interpretive error: treating every low-rank gradient matrix as evidence that examples agree on direction. This is the paper’s most consequential contribution.
- A coherent, multi-stage diagnostic study. The authors connect the local geometry to token-level activation/backpropagation pairing and test the distributional pattern across models, stages, datasets, and independently sampled minibatches. This gives the descriptive claim more support than a single-checkpoint case study.
Critical weaknesses
- The optimization experiment does not isolate the contribution of NEER-based selection. The paper compares selective local clipping with a baseline, but not with standard global/parameter-wise clipping or a control that clips the same number of sequence contributions in randomly selected columns. Thus, the experiment supports the plausibility of local rebalancing, while leaving open whether NEER-based targeting is necessary for the observed gains. This limits the intervention claim, not the paper’s principal diagnostic contribution.
- The scope and operating cost of the intervention remain narrow. The downstream intervention is tested on mathematical fine-tuning and four math benchmarks, with mixed per-benchmark outcomes for Mistral-7B; the method also needs explicit per-sequence gradients and local NEER computation, whose overhead is not quantified. The authors appropriately frame it as a mechanism test and note the need for efficient surrogates, but the current evidence does not establish a practical optimizer recipe.
Questions for authors
Clarification requests
- Did you evaluate a matched standard-clipping control (global norm clipping, per-parameter clipping, or clipping the same number of top sequence contributions in randomly selected columns)? Such a control would clarify whether NEER-based local selection is responsible for the improvement beyond generic clipping.
- What is the additional wall-clock time and peak memory for computing per-sequence, per-column gradients and applying the intervention, relative to the baseline training setup?
- How sensitive are the downstream results to the intervention settings (the selected column fraction and number of clipped sequences per column), given that the paper uses and by default?
Suggestions for future work
- A lower-cost online proxy for identifying local magnitude dominance, evaluated against the explicit per-sequence-gradient method, would help determine whether the proposed mechanism can translate into a practical training algorithm.
Clarification of the motivation and the term “energy”
We will revise the introduction to explain our research motivation and the choice of NEER more clearly, as outlined in our response to all reviewers above. We will also define terminology before using it. In particular, the term energy means the squared Euclidean norm of a vector in this paper. This terminology follows the established squared-norm convention in signal processing and related mathematical treatments, and gives us a concise way to discuss both sample-level and spectral contributions. We will state this definition explicitly at its first use.