- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how RMSNorm affects the theoretical complexity and generalization of transformer models. It shows that RMSNorm can increase Rademacher complexity even when the hidden state variation is small and gives a worst-case bound depending on sample count, dimension, and radius variation. Experimentally, the theoretical worst-case behavior does not actually worsen generalization in pertrained LLMs.
Critical strengths
- The paper provides the theoretical insight that RMSNorm can increase worst-case model complexity, even when hidden-state magnitudes vary only slightly.
- The paper stresses the distinct roles of hidden-state radius variation and radius level, and shows how architectural constraints limit the worst-case complexity increase.
Critical weaknesses
- The paper is difficult to follow due to insufficient background explanations and intuition, especially in the abstract and introduction. For example, notations such as , , , and are used before being clearly defined. Furthermore, the intuition behind the synthetic construction is not sufficiently explained, making its purpose difficult to understand.
- The experiments evaluate Rademacher complexity distortion before and after RMSNorm only on its synthetic model classes (). Although pretrained LLM checkpoints are examined for hidden-state radii and their theoretical guarantees, the paper does not measure this distortion on realistic model neighborhoods.
Questions for authors
- Could the authors introduce the key notations earlier and provide more intuition for the synthetic construction, particularly why it is chosen, why this can represent hidden-state variation faithfully, and how it demonstrates normalization-induced complexity distortion?
- Could the authors evaluate the Rademacher complexity distortion before and after RMSNorm on realistic pretrained model neighborhoods, for example, by perturbing upstream model parameters to induce realistic hidden-state variation?
7 comments
Response to 6cxz (2/4): Complexity measurements in pretrained neighborhoods
2. Weakness 2 and Question 2: pretrained-neighborhood complexity
Weakness 2
The experiments evaluate Rademacher complexity distortion before and after RMSNorm only on its synthetic model classes (). Although pretrained LLM checkpoints are examined for hidden-state radii and their theoretical guarantees, the paper does not measure this distortion on realistic model neighborhoods.
Question 2
Could the authors evaluate the Rademacher complexity distortion before and after RMSNorm on realistic pretrained model neighborhoods, for example, by perturbing upstream model parameters to induce realistic hidden-state variation?
Response. Section 5.1 performs this evaluation on SmolLM2-135M, SmolLM2-135M-Instruct, and SmolLM2-360M. We perturb every entry of the final MLP down projection in complete Frobenius and spectral balls. This matrix is upstream of the final residual addition and RMSNorm. With the upstream model frozen,
This identity is exact, and a direct intervention on the model matrix checks it against the forward pass. Features come from WikiText-103 contexts of length 128, using their final position. All feature components, output coordinates, and matrix entries are retained. The base-135M run confirms geometry using fresh sign draws on the pilot inputs; the instruct and 360M runs extract new features and use matched recorded inputs. All 48 planned settings are complete.
What is calculated
Theorem 4.4 gives the exact raw inner supremum for every sign vector. A frozen-denominator control divides each perturbed state by that input's reference RMS denominator, retaining the same parameter ball. Proposition O.10 gives its exact inner supremum as well. We estimate the outer sign expectations using 65,536 reference draws.
For normalized complexity, optimized feasible perturbations supply lower witnesses to the true supremum. Each positive primary setting uses 2,048 optimized sign draws, with 80 search steps and three starts per draw, including the zero perturbation. The paired confidence analysis converts these witnesses into lower bounds on the difference from the exact frozen control.
Let C_raw and C_fr denote raw and frozen-control complexity, respectively, and C_N denotes true normalized complexity for the selected matrix norm. Hats indicate numerical estimates; the hatted W_F and W_op values estimate the feasible-witness expectations for the Frobenius and spectral balls, respectively. The table reports all positive-budget primary comparisons at m = 64. For each checkpoint, the neighborhood radius is the budget fraction f multiplied by the minimum ratio of the reference hidden-state Euclidean norm to the projection-input Euclidean norm across the calibration inputs. Using the same radius gives the two norm balls the same maximum pointwise displacement.
| Checkpoint | f | Raw complexity estimate | Frozen-control estimate | Frobenius witness estimate | Spectral witness estimate | Difference lower bound |
|---|---|---|---|---|---|---|
| 135M | 0.15 | 120.404 | 3.250 | 3.194 | 3.700 | 0.269 |
| 135M | 0.30 | 133.324 | 3.602 | 3.490 | 4.635 | 0.620 |
| 135M-Instruct | 0.15 | 122.709 | 3.165 | 3.096 | 3.584 | 0.249 |
| 135M-Instruct | 0.30 | 135.468 | 3.495 | 3.357 | 4.445 | 0.568 |
| 360M | 0.15 | 190.820 | 4.109 | 3.982 | 4.489 | 0.149 |
| 360M | 0.30 | 209.482 | 4.513 | 4.268 | 5.394 | 0.358 |
Response table C1. Before-and-after complexity in complete pretrained projection neighborhoods at . is a simultaneous confidence lower bound on . Raw and frozen complexity are the same for the two norm types; normalized witness values depend on the norm.
A certified changing-denominator effect
All six bounds in the last column are positive. Thus true normalized spectral complexity exceeds the frozen-denominator control in every positive-budget primary setting. For each sign draw, the normalized supremum is at least the value of its feasible witness, so a lower confidence bound on the expected witness-minus-control difference also bounds from below.
Proposition O.10 formalizes this comparison. The joint allocation provides at least 95% coverage for the geometry and radius-validation families across the three runs. Geometry confidence statements quantify sign-integration uncertainty conditional on the recorded inputs.
Response to 6cxz (3/4): Frozen-control and floor-adjusted measurements
Continuation of Weakness 2 and Question 2
Two complementary measurements
Alongside the frozen-denominator comparison, we report the floor-adjusted quantity corresponding to the paper's contraction analysis:
where is the exact minimum radius over the complete parameter ball and geometry sample. At :
| Checkpoint | Frozen-control | Witness-based estimate | Simultaneous interval for | |
|---|---|---|---|---|
| 135M | 1.287 | 0.360 | 0.463 | |
| 135M-Instruct | 1.272 | 0.406 | 0.516 | |
| 360M | 1.195 | 0.470 | 0.562 |
Response table C2. Frozen-control and floor-adjusted measurements at , . Frozen-control applies the same floor adjustment to the frozen-denominator control. The witness-based estimate uses the feasible-witness complexity in the numerator. The simultaneous intervals for true combine witness lower bounds, analytic upper bounds, and numerical uncertainty.
How the measurements connect
Applying the same complete-ball, geometry-sample floor to the frozen control gives 0.360, 0.406, and 0.470. Multiplying these values by the witness/control ratios in the first column gives the witness-based estimates in the third column. The table therefore expresses the measured witnesses both relative to the frozen control and on the floor-adjusted scale.
The frozen control retains each input's reference scaling and the complete parameter ball. Allowing the denominator to vary with the perturbed projection produces the positive simultaneous complexity differences reported in Part 2. This comparison directly quantifies the effect of hypothesis-dependent normalization at trained checkpoints.
Complete experimental reporting
The primary comparison uses the prespecified sample size . Appendix L.4 reports all 48 settings, including the additional sample sizes and , with witness estimates and confidence bounds.
The descriptive witness/control estimates at increase across : 1.216, 1.272, 1.330 for 135M-Instruct, and 1.159, 1.195, 1.240 for 360M. The complete tables provide the corresponding numerical and confidence information for each setting.
Certified floors and prediction changes
These same neighborhoods have positive parameter-uniform radius floors across the validation inputs. At the largest budget, validation minima are 13.058, 12.216, and 16.442 for the three checkpoints. The two new checkpoints have zero target-floor failures in 1,024 validation draws each, with probability bounds under the recorded finite-pool distributions.
Prediction diagnostics retain the trained RMSNorm gain and vocabulary head. Prespecified spectral witnesses change hidden states by about 20–21% and next-token argmax predictions on about 23–24% of contexts. Thus the neighborhoods used for the complexity measurements contain consequential changes to the predictor.
The geometry experiment evaluates the gain-free normalizer studied in the theory within complete neighborhoods of one final projection. The prediction diagnostics then connect these perturbations to the full trained output map. Together, the synthetic studies test the constructed scaling law, and the SmolLM2 experiments measure the denominator effect, radius certificates, and predictor changes at actual checkpoints.
Changes in the revised manuscript: Theorem 4.4 (p. 8); Section 5.1 (pp. 9–10), Tables 2 and 3 (pp. 9–10), and Figure 2 in Appendix L (p. 40); paired guarantee in Proposition O.10 (p. 50); full protocol in Appendix L.
Response to 6cxz (1/4): Notation and intuition behind the construction
Thank you for asking for a clearer explanation of the construction and a direct experiment in pretrained parameter neighborhoods. The revision introduces the notation before the overview and explains the normalization mechanism beside Figure 1. The new experiments evaluate complete neighborhoods at three pretrained checkpoints and establish positive simultaneous lower bounds for normalized spectral complexity relative to a frozen-denominator control. We describe the exposition changes and experimental evidence below.
1. Weakness 1 and Question 1: notation and construction intuition
Weakness 1
Question 1
Response. The abstract now identifies m as the number of evaluation inputs, d as the hidden width, and ν>0 as the RMS denominator regularizer. The introduction defines the RMS radius ∥h∥∘=∥h∥2/d, and Section 2 introduces the remaining notation. The setup explains the complexity process through a bounded scalar readout: optimizing the readout over a Euclidean ball gives the Euclidean norm of the signed hidden-state sum, with one independent sign per input.
Figure 1 now accompanies the following mechanism explanation. Before normalization, every hi(a)=d(e0+aiei) has the same e0 coordinate, while changes in different ei directions remain orthogonal. After normalization, choosing ai changes the radius and rescales the shared e0 coordinate. Choosing the smaller radius on positive signs and the larger radius on negative signs makes contributions in the shared direction add coherently. The orthogonal coordinates allow the denominator to change while keeping the corresponding raw perturbations orthogonal.
Each vector a specifies one shared hypothesis on all inputs; the Rademacher supremum selects it after the signs are observed. This explains how the construction isolates the effect of a hypothesis-dependent denominator and attains the sharp distortion regimes.
The revision also connects the construction to complete parameter classes. Remark 3.5 states the matrix-norm distinction, Section 4.4 explains it, and Appendix O gives the full results: many orthogonal directions can simultaneously use the spectral budget, whereas their squared sizes accumulate under a Frobenius budget.
Theorem O.4 and Corollary O.5 extend the sharp regimes to a complete spectral projection ball, including its full raw denominator. Corollary O.3 gives the contrasting dimension-independent bound for a complete affine Frobenius ball satisfying the common RMS annulus [1,1+2ρ] throughout the ball at every evaluated input. Section 5.1 then evaluates complete neighborhoods using actual pretrained features. These additions connect the construction's mechanism, the geometry of full parameter balls, and the trained-model experiments.
Changes in the revised manuscript: Abstract/introduction (p. 1); Section 2 (p. 3); Section 3 and Figure 1 (pp. 3–4); complete-ball results in Remark 3.5 (p. 5), with full statements and proofs in Appendix O. The manuscript retains ten main-text pages.