acceptodds
Under review as a conference paper at ICLR 2027

Radius Level and Radius Variation in Pre-Normalized Decoders: Sharp Normalization Distortion and Width Dependence

Abstract

We study the worst-case cost of RMS normalization in the single-sign vector Rademacher process—one Rademacher sign per example, a Euclidean norm outside the signed sum—that arises when a bounded linear readout is peeled from a hidden-state class. We separate out two effects : variation of hidden-state radii across hypotheses, and distance of those states from zero. For fixed and , on the annulus with , the dimension-unconstrained sharp (with matching lower bound) order is ; at every fixed positive  the width-aware sharp order is , so distortion saturates at in a fixed-width architecture. A uniform joint envelope for bounded  identifies the remaining gap; it matches throughout and . Conditionally on a bound for the raw hidden-state complexity, these factors enter a standard bounded-linear-readout generalization inequality. For radius level, we derive sufficient residual-amplitude conditions under which an embedding floor propagates to a population hidden-state floor at every depth, and give explicit attention and SwiGLU amplitude bounds. A five-checkpoint audit on public pre-RMS decoders shows that the tested uniform certificates fail at native scale, although sampled radii remain positive throughout depth.

Primary area:
learning theory
Keywords:
pre-normalized transformers, RMS normalization, Rademacher complexity, norm-based generalization, hidden-state norms, residual networks, transformer theory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6cxzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how RMSNorm affects the theoretical complexity and generalization of transformer models. It shows that RMSNorm can increase Rademacher complexity even when the hidden state variation is small and gives a worst-case bound depending on sample count, dimension, and radius variation. Experimentally, the theoretical worst-case behavior does not actually worsen generalization in pertrained LLMs.

Critical strengths
  • The paper provides the theoretical insight that RMSNorm can increase worst-case model complexity, even when hidden-state magnitudes vary only slightly.
  • The paper stresses the distinct roles of hidden-state radius variation and radius level, and shows how architectural constraints limit the worst-case complexity increase.
Critical weaknesses
  • The paper is difficult to follow due to insufficient background explanations and intuition, especially in the abstract and introduction. For example, notations such as mm, dd, ν\nu, and ∥h∥∘\|h\|_\circ are used before being clearly defined. Furthermore, the intuition behind the synthetic construction hi(a)=d(e0+aiei)h_i(a)=\sqrt{d}(e_0+a_i e_i) is not sufficiently explained, making its purpose difficult to understand.
  • The experiments evaluate Rademacher complexity distortion before and after RMSNorm only on its synthetic model classes (ai∈{0,t}a_i\in\{0,t\}). Although pretrained LLM checkpoints are examined for hidden-state radii and their theoretical guarantees, the paper does not measure this distortion on realistic model neighborhoods.
Questions for authors
  • Could the authors introduce the key notations earlier and provide more intuition for the synthetic construction, particularly why it is chosen, why this can represent hidden-state variation faithfully, and how it demonstrates normalization-induced complexity distortion?
  • Could the authors evaluate the Rademacher complexity distortion before and after RMSNorm on realistic pretrained model neighborhoods, for example, by perturbing upstream model parameters to induce realistic hidden-state variation?
7 comments
AuthorsOpenReview ↗

Response to 6cxz (1/4): Notation and intuition behind the construction

Thank you for asking for a clearer explanation of the construction and a direct experiment in pretrained parameter neighborhoods. The revision introduces the notation before the overview and explains the normalization mechanism beside Figure 1. The new experiments evaluate complete neighborhoods at three pretrained checkpoints and establish positive simultaneous lower bounds for normalized spectral complexity relative to a frozen-denominator control. We describe the exposition changes and experimental evidence below.

1. Weakness 1 and Question 1: notation and construction intuition

Weakness 1

The paper is difficult to follow due to insufficient background explanations and intuition, especially in the abstract and introduction. For example, notations such as mm, dd, ν\nu, and ∥h∥∘\lVert h\rVert_\circ are used before being clearly defined. Furthermore, the intuition behind the synthetic construction hi(a)=d(e0+aiei)h_i(a)=\sqrt{d}(e_0+a_i e_i) is not sufficiently explained, making its purpose difficult to understand.

Question 1

Could the authors introduce the key notations earlier and provide more intuition for the synthetic construction, particularly why it is chosen, why this can represent hidden-state variation faithfully, and how it demonstrates normalization-induced complexity distortion?

Response. The abstract now identifies mm as the number of evaluation inputs, dd as the hidden width, and ν>0\nu > 0 as the RMS denominator regularizer. The introduction defines the RMS radius ∥h∥∘=∥h∥2/d\lVert h\rVert_\circ=\lVert h\rVert_2/\sqrt{d}, and Section 2 introduces the remaining notation. The setup explains the complexity process through a bounded scalar readout: optimizing the readout over a Euclidean ball gives the Euclidean norm of the signed hidden-state sum, with one independent sign per input.

Figure 1 now accompanies the following mechanism explanation. Before normalization, every hi(a)=d(e0+aiei)h_i(a)=\sqrt{d}(e_0+a_i e_i) has the same e0e_0 coordinate, while changes in different eie_i directions remain orthogonal. After normalization, choosing aia_i changes the radius and rescales the shared e0e_0 coordinate. Choosing the smaller radius on positive signs and the larger radius on negative signs makes contributions in the shared direction add coherently. The orthogonal coordinates allow the denominator to change while keeping the corresponding raw perturbations orthogonal.

Each vector aa specifies one shared hypothesis on all inputs; the Rademacher supremum selects it after the signs are observed. This explains how the construction isolates the effect of a hypothesis-dependent denominator and attains the sharp distortion regimes.

The revision also connects the construction to complete parameter classes. Remark 3.5 states the matrix-norm distinction, Section 4.4 explains it, and Appendix O gives the full results: many orthogonal directions can simultaneously use the spectral budget, whereas their squared sizes accumulate under a Frobenius budget.

Theorem O.4 and Corollary O.5 extend the sharp regimes to a complete spectral projection ball, including its full raw denominator. Corollary O.3 gives the contrasting dimension-independent bound for a complete affine Frobenius ball satisfying the common RMS annulus [1,1+2ρ][1,1+2\rho] throughout the ball at every evaluated input. Section 5.1 then evaluates complete neighborhoods using actual pretrained features. These additions connect the construction's mechanism, the geometry of full parameter balls, and the trained-model experiments.

Changes in the revised manuscript: Abstract/introduction (p. 1); Section 2 (p. 3); Section 3 and Figure 1 (pp. 3–4); complete-ball results in Remark 3.5 (p. 5), with full statements and proofs in Appendix O. The manuscript retains ten main-text pages.

AuthorsOpenReview ↗

Response to 6cxz (2/4): Complexity measurements in pretrained neighborhoods

2. Weakness 2 and Question 2: pretrained-neighborhood complexity

Weakness 2

The experiments evaluate Rademacher complexity distortion before and after RMSNorm only on its synthetic model classes (ai∈{0,t}a_i\in\lbrace 0,t\rbrace). Although pretrained LLM checkpoints are examined for hidden-state radii and their theoretical guarantees, the paper does not measure this distortion on realistic model neighborhoods.

Question 2

Could the authors evaluate the Rademacher complexity distortion before and after RMSNorm on realistic pretrained model neighborhoods, for example, by perturbing upstream model parameters to induce realistic hidden-state variation?

Response. Section 5.1 performs this evaluation on SmolLM2-135M, SmolLM2-135M-Instruct, and SmolLM2-360M. We perturb every entry of the final MLP down projection in complete Frobenius and spectral balls. This matrix is upstream of the final residual addition and RMSNorm. With the upstream model frozen,

hΔ(x)=h0(x)+z(x)Δ,∥Δ∥a≤η,a∈{F,op}.h_\Delta(x)=h_0(x)+z(x)\Delta,\qquad\lVert\Delta\rVert_{\mathsf{a}}\le\eta,\qquad\mathsf{a}\in\lbrace F,\mathrm{op}\rbrace.

This identity is exact, and a direct intervention on the model matrix checks it against the forward pass. Features come from WikiText-103 contexts of length 128, using their final position. All feature components, output coordinates, and matrix entries are retained. The base-135M run confirms geometry using fresh sign draws on the pilot inputs; the instruct and 360M runs extract new features and use matched recorded inputs. All 48 planned settings are complete.

What is calculated

Theorem 4.4 gives the exact raw inner supremum for every sign vector. A frozen-denominator control divides each perturbed state by that input's reference RMS denominator, retaining the same parameter ball. Proposition O.10 gives its exact inner supremum as well. We estimate the outer sign expectations using 65,536 reference draws.

For normalized complexity, optimized feasible perturbations supply lower witnesses to the true supremum. Each positive primary setting uses 2,048 optimized sign draws, with 80 search steps and three starts per draw, including the zero perturbation. The paired confidence analysis converts these witnesses into lower bounds on the difference from the exact frozen control.

Let C_raw and C_fr denote raw and frozen-control complexity, respectively, and C_N denotes true normalized complexity for the selected matrix norm. Hats indicate numerical estimates; the hatted W_F and W_op values estimate the feasible-witness expectations for the Frobenius and spectral balls, respectively. The table reports all positive-budget primary comparisons at m = 64. For each checkpoint, the neighborhood radius is the budget fraction f multiplied by the minimum ratio of the reference hidden-state Euclidean norm to the projection-input Euclidean norm across the calibration inputs. Using the same radius gives the two norm balls the same maximum pointwise displacement.

CheckpointfRaw complexity estimateFrozen-control estimateFrobenius witness estimateSpectral witness estimateDifference lower bound
135M0.15120.4043.2503.1943.7000.269
135M0.30133.3243.6023.4904.6350.620
135M-Instruct0.15122.7093.1653.0963.5840.249
135M-Instruct0.30135.4683.4953.3574.4450.568
360M0.15190.8204.1093.9824.4890.149
360M0.30209.4824.5134.2685.3940.358

Response table C1. Before-and-after complexity in complete pretrained projection neighborhoods at m=64m=64. LopL_{\mathrm{op}} is a simultaneous confidence lower bound on CNop−CfrC_N^{\mathrm{op}}-C_{\mathrm{fr}}. Raw and frozen complexity are the same for the two norm types; normalized witness values depend on the norm.

A certified changing-denominator effect

All six bounds in the last column are positive. Thus true normalized spectral complexity exceeds the frozen-denominator control in every positive-budget primary setting. For each sign draw, the normalized supremum is at least the value of its feasible witness, so a lower confidence bound on the expected witness-minus-control difference also bounds CNop−CfrC_N^{\mathrm{op}}-C_{\mathrm{fr}} from below.

Proposition O.10 formalizes this comparison. The joint allocation provides at least 95% coverage for the geometry and radius-validation families across the three runs. Geometry confidence statements quantify sign-integration uncertainty conditional on the recorded inputs.

AuthorsOpenReview ↗

Response to 6cxz (3/4): Frozen-control and floor-adjusted measurements

Continuation of Weakness 2 and Question 2

Two complementary measurements

Alongside the frozen-denominator comparison, we report the floor-adjusted quantity corresponding to the paper's contraction analysis:

DSop=cS2+ν CNopCraw,D_S^{\mathrm{op}}=\sqrt{c_S^2+\nu}\,\frac{C_N^{\mathrm{op}}}{C_{\mathrm{raw}}},

where cSc_S is the exact minimum radius over the complete parameter ball and geometry sample. At f=0.30f=0.30:

CheckpointW^op/C^fr\widehat{W}_{\mathrm{op}}/\widehat{C}_{\mathrm{fr}}Frozen-control DDWitness-based DD estimateSimultaneous interval for DD
135M1.2870.3600.463[0.393,2.580][0.393, 2.580]
135M-Instruct1.2720.4060.516[0.430,3.005][0.430, 3.005]
360M1.1950.4700.562[0.458,3.485][0.458, 3.485]

Response table C2. Frozen-control and floor-adjusted measurements at m=64m=64, f=0.30f=0.30. Frozen-control DD applies the same floor adjustment to the frozen-denominator control. The witness-based estimate uses the feasible-witness complexity in the numerator. The simultaneous intervals for true DD combine witness lower bounds, analytic upper bounds, and numerical uncertainty.

How the measurements connect

Applying the same complete-ball, geometry-sample floor cSc_S to the frozen control gives 0.360, 0.406, and 0.470. Multiplying these values by the witness/control ratios in the first column gives the witness-based estimates in the third column. The table therefore expresses the measured witnesses both relative to the frozen control and on the floor-adjusted scale.

The frozen control retains each input's reference scaling and the complete parameter ball. Allowing the denominator to vary with the perturbed projection produces the positive simultaneous complexity differences reported in Part 2. This comparison directly quantifies the effect of hypothesis-dependent normalization at trained checkpoints.

Complete experimental reporting

The primary comparison uses the prespecified sample size m=64m=64. Appendix L.4 reports all 48 settings, including the additional sample sizes m=32m=32 and 128128, with witness estimates and confidence bounds.

The descriptive witness/control estimates at f=0.30f=0.30 increase across m=32,64,128m=32,64,128: 1.216, 1.272, 1.330 for 135M-Instruct, and 1.159, 1.195, 1.240 for 360M. The complete tables provide the corresponding numerical and confidence information for each setting.

Certified floors and prediction changes

These same neighborhoods have positive parameter-uniform radius floors across the validation inputs. At the largest budget, validation minima are 13.058, 12.216, and 16.442 for the three checkpoints. The two new checkpoints have zero target-floor failures in 1,024 validation draws each, with probability bounds under the recorded finite-pool distributions.

Prediction diagnostics retain the trained RMSNorm gain and vocabulary head. Prespecified spectral witnesses change hidden states by about 20–21% and next-token argmax predictions on about 23–24% of contexts. Thus the neighborhoods used for the complexity measurements contain consequential changes to the predictor.

The geometry experiment evaluates the gain-free normalizer studied in the theory within complete neighborhoods of one final projection. The prediction diagnostics then connect these perturbations to the full trained output map. Together, the synthetic studies test the constructed scaling law, and the SmolLM2 experiments measure the denominator effect, radius certificates, and predictor changes at actual checkpoints.

Changes in the revised manuscript: Theorem 4.4 (p. 8); Section 5.1 (pp. 9–10), Tables 2 and 3 (pp. 9–10), and Figure 2 in Appendix L (p. 40); paired guarantee in Proposition O.10 (p. 50); full protocol in Appendix L.

Reviewer_JppcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how RMS normalization affects the complexity of hidden-state classes in pre-normalized decoders. It distinguishes radius variation across hypotheses from radius level, meaning a uniform lower bound on hidden-state norms. For the single-sign vector Rademacher complexity, it establishes matching distortion bounds when width is unrestricted and, separately, when the annulus width is fixed and positive. The width-dependent analysis shows that the worst-case distortion saturates at fixed dimension, while identifying a remaining gap in the joint dependence on annulus width, sample size, and dimension. The paper also derives sufficient radius-floor conditions from cumulative residual-branch amplitudes, with directional and sensitivity refinements. Validation combines synthetic constructions with audits of five public decoder checkpoints. The constructions illustrate the theoretical scaling, whereas the checkpoint audits find positive sampled radii but fail to establish positive floors using the tested uniform sufficient conditions.

Critical strengths

The distinction between radius variation and radius level is useful, and the width-dependent analysis is substantive. Matching upper and lower bounds in the stated regimes clarify when normalization distortion can grow with sample size and when dimension limits that growth. The explicit constructions make these worst-case effects concrete.

Critical weaknesses
  1. The generalization interpretation contains a dimensional-scaling inconsistency. Section 6 states that construction (9), with d=m+1d=m+1, has raw complexity of order m−1/2m^{-1/2}. However, Equation (39) gives Rmvec=Θ(d/m)\mathfrak{R}^{\mathrm{vec}}_m=\Theta(\sqrt{d/m}) for fixed positive annulus width, which is Θ(1)\Theta(1) in this setting under the stated Euclidean definition. An additional dimension normalization or dimension-dependent readout budget appears necessary for the claimed decay. This does not invalidate the distortion-ratio results, but it weakens the interpretation that normalization turns vanishing raw complexity into nonvanishing complexity. Moreover, the generalization bound remains conditional on controlling the raw hidden-state complexity.

  2. The practical significance of the radius-floor certificates remains limited. All tested isotropic and directional sufficient floors fail at native checkpoint scales (Table 5), with the amplitude diagnostics exceeding the admissible scale by many orders of magnitude. Positive sampled radii do not establish the uniform population conditions required by the theory. Likewise, variation across tokens at a fixed checkpoint does not measure variation across hypotheses at a fixed input. The audits honestly expose these limitations, but they do not demonstrate a useful certificate for a trained model or a meaningful parameter neighborhood, leaving a substantial gap between the theoretical guarantees and their application to decoders.

Questions for authors
  1. How should the m−1/2m^{-1/2} raw-complexity claim in Section 6 be reconciled with Equation (39) when d=m+1d=m+1? Is an additional normalization or readout-budget assumption intended?
  2. Is there a nontrivial trained-model setting or parameter neighborhood where the proposed sufficient conditions yield a positive, useful radius floor? Such evidence would materially strengthen the practical significance of the results.
6 comments
AuthorsOpenReview ↗

Response to Jppc (1/3): Scaling correction and explicit raw complexity

Thank you for your detailed questions on dimensional scaling, raw complexity, and trained-model certificates. They led to a precise readout interpretation, an exact complexity formula for complete projection neighborhoods, and positive local certificates at three pretrained checkpoints. We address each concern below.

1. Weakness 1 and Question 1: dimensional scaling

Weakness 1 (scaling concern)

The generalization interpretation contains a dimensional-scaling inconsistency. Section 6 states that construction (9), with d=m+1d=m+1, has raw complexity of order m−1/2m^{-1/2}. However, Equation (39) gives Rmvec=Θ(d/m)\mathfrak{R}_m^{\mathrm{vec}}=\Theta(\sqrt{d/m}) for fixed positive annulus width, which is Θ(1)\Theta(1) in this setting under the stated Euclidean definition. An additional dimension normalization or dimension-dependent readout budget appears necessary for the claimed decay. This does not invalidate the distortion-ratio results, but it weakens the interpretation that normalization turns vanishing raw complexity into nonvanishing complexity.

Question 1

How should the m−1/2m^{-1/2} raw-complexity claim in Section 6 be reconciled with Equation (39) when d=m+1d=m+1? Is an additional normalization or readout-budget assumption intended?

Response. We have corrected the discussion and mechanism caption to use the dimensionally correct expression throughout. Under the Euclidean vector definition, the raw complexity is Θ(d/m)\Theta(\sqrt{d/m}), hence Θ(1)\Theta(1) when d=m+1d=m+1. For Sm=∑iεiS_m=\sum_i\varepsilon_i and t2=4ρ(1+ρ)t^2=4\rho(1+\rho), the exact expression is

Rmvec(H)=dmESm2+t2m.\mathfrak{R}_m^{\mathrm{vec}}(\mathcal{H})=\frac{\sqrt{d}}{m}\mathbb{E}\sqrt{S_m^2+t^2m}.

At fixed positive ρ\rho and ν\nu, the corresponding normalized complexity is Θ(d)\Theta(\sqrt{d}). The distortion-ratio results retain their statements, and their proofs already use this raw expression.

Corollary 3.4 states the scalar capacity separation under the explicit, common readout constraint ∥w∥2≤B0/d\lVert w\rVert_2\le B_0/\sqrt{d}. For the raw and normalized scalar classes, respectively,

Rm(Fraw)=Θ(B0m−1/2),Rm(FN)=Θ(B0).\mathfrak{R}_m(\mathcal{F}_{\mathrm{raw}})=\Theta(B_0m^{-1/2}),\qquad \mathfrak{R}_m(\mathcal{F}_N)=\Theta(B_0).

The constants may depend on the fixed ρ,ν\rho,\nu. The same readout budget applies to both classes and gives width-independent prediction bounds: B0(1+2ρ)B_0(1+2\rho) for raw predictions and B0B_0 for normalized predictions. This makes the readout assumption and the resulting scalar separation explicit.

Changes in the revised manuscript: Section 3, Corollary 3.4 (p. 5), and Figure 1 (p. 4); interpretation in Section 6 (p. 10).

2. Remaining part of Weakness 1: the raw complexity term

Weakness 1 (raw-complexity concern)

Moreover, the generalization bound remains conditional on controlling the raw hidden-state complexity.

Response. Theorem 4.4 gives an explicit expression for this term in a complete local decoder class. We freeze the upstream model and vary every entry of its final residual projection in a complete Frobenius or spectral ball of radius η\eta. The hidden states satisfy hi(Δ)=hi0+ziΔh_i(\Delta)=h_i^0+z_i\Delta exactly, where ziz_i is the projection input. Then

Craw:=Rmvec(Hη,S)=1mEε[∥∑i=1mεihi0∥2+η∥∑i=1mεizi∥2].C_{\mathrm{raw}}:=\mathfrak{R}_m^{\mathrm{vec}}(\mathcal{H}_{\eta,S})=\frac{1}{m}\mathbb{E}_{\varepsilon}\left[\left\lVert\sum_{i=1}^m\varepsilon_i h_i^0\right\rVert_2+\eta\left\lVert\sum_{i=1}^m\varepsilon_i z_i\right\rVert_2\right].

This formula optimizes over the entire ball under either norm, with an exact inner supremum for each sign vector. Appendix O.1 gives explicit moment bounds. Theorem O.6 incorporates these quantities into a local generalization inequality for a class fixed independently of the i.i.d. evaluation sample. Corollary O.7 gives an m−1/2m^{-1/2} Frobenius consequence under its uniform input-support radius and displacement assumptions and the budget B0/dB_0/\sqrt{d}.

The reference hidden states, projection features, and neighborhood radius therefore determine the raw-complexity term appearing in the local bound.

Changes in the revised manuscript: Theorem 4.4 (p. 8) and Section 4.4 (pp. 7–8); full generalization statements in Theorem O.6 and Corollary O.7 (p. 48).

AuthorsOpenReview ↗

Response to Jppc (2/3): Positive radius certificates at pretrained checkpoints

3. Weakness 2 and Question 2: a useful trained-model radius certificate

Weakness 2

The practical significance of the radius-floor certificates remains limited. All tested isotropic and directional sufficient floors fail at native checkpoint scales (Table 5), with the amplitude diagnostics exceeding the admissible scale by many orders of magnitude. Positive sampled radii do not establish the uniform population conditions required by the theory. Likewise, variation across tokens at a fixed checkpoint does not measure variation across hypotheses at a fixed input. The audits honestly expose these limitations, but they do not demonstrate a useful certificate for a trained model or a meaningful parameter neighborhood, leaving a substantial gap between the theoretical guarantees and their application to decoders.

Question 2

Is there a nontrivial trained-model setting or parameter neighborhood where the proposed sufficient conditions yield a positive, useful radius floor? Such evidence would materially strengthen the practical significance of the results.

Response. The new local result, Theorem 4.4, provides the requested trained-model certificate for complete neighborhoods around SmolLM2-135M, SmolLM2-135M-Instruct, and SmolLM2-360M. We vary every entry of the final MLP down-projection matrix while fixing the upstream model. The final pre-RMSNorm state is exactly affine in this matrix, giving the exact minimum at each input:

gη(x):=inf⁡∥Δ∥a≤η∥h0(x)+z(x)Δ∥2d=(∥h0(x)∥2−η∥z(x)∥2)+d,a∈{F,op}.g_\eta(x):=\inf_{\lVert\Delta\rVert_{\mathsf{a}}\le\eta}\frac{\lVert h_0(x)+z(x)\Delta\rVert_2}{\sqrt{d}}=\frac{(\lVert h_0(x)\rVert_2-\eta\lVert z(x)\rVert_2)_+}{\sqrt{d}},\quad\mathsf{a}\in\lbrace F,\mathrm{op}\rbrace.

The infimum covers every matrix in the ball, directly capturing across-hypothesis behavior at a fixed input. Separate calibration draws determine the budget and target floor:

η=fη⋆,η⋆=min⁡cal∥h0(x)∥2∥z(x)∥2,c⋆=14min⁡cal∥h0(x)∥2d.\eta=f\eta_\star,\qquad\eta_\star=\min_{\mathrm{cal}}\frac{\lVert h_0(x)\rVert_2}{\lVert z(x)\rVert_2},\qquad c_\star=\frac{1}{4}\min_{\mathrm{cal}}\frac{\lVert h_0(x)\rVert_2}{\sqrt{d}}.

CheckpointTarget c⋆c_\starMinimum certified floorFailures / drawsFailure-probability upper bound
135M5.79913.0580/2562.3828%
135M-Instruct5.56912.2160/1,0240.6011%
360M6.86016.4420/1,0240.6011%

Response table J1. Local radius certificates at the largest tested fraction, f=0.30f=0.30, for SmolLM2 checkpoints. The minimum is taken over the validation inputs after minimizing over the entire parameter ball. A failure means gη(X)<c⋆g_\eta(X) < c_\star.

The two new checkpoint runs have zero failures in 1,024 draws each, sampled independently with replacement from their recorded finite test-context pools. Their simultaneous failure-probability upper bound is 0.6011%.

The base-135M analysis uses its 256 pilot validation draws. Its budget fractions and target fraction were fixed in the pilot code before those draws were inspected, and its numerical target comes from 32 separate calibration draws. The confirmation analysis combines this same validation sample with fresh geometry sign draws under the joint confidence allocation, giving the certificate bound of 2.3828%.

All four prespecified certificate budgets have zero observed failures in each model. The confidence allocation covers all reported certificate budgets and geometry comparisons, with at least 95% joint coverage. The validation probability bounds apply to the recorded finite-pool input distributions, and the radius calculation covers every parameter in each ball.

These results supply positive local floors at trained checkpoints with independently calibrated budgets and a complete parameter-neighborhood guarantee at every validation input.

AuthorsOpenReview ↗

Response to Jppc (3/3): Predictor changes, certificate interpretation, and presentation

Continuation of Weakness 2 and Question 2

Why these neighborhoods are informative

At f=0.30f=0.30, the budgets are 8.89%, 9.19%, and 5.77% of the trained projections' spectral norms and 0.94%, 0.97%, and 0.62% of their Frobenius norms, respectively, for 135M, 135M-Instruct, and 360M. The certificate covers every matrix in these balls, with validation on inputs separate from calibration.

We also evaluate the first eight prespecified complexity witnesses per setting on 256 separate prediction draws, retaining the trained RMSNorm gain and vocabulary head:

CheckpointNormHidden change (%)Argmax change (%)KL (nats)Δ\DeltaNLL
135M-InstructFrobenius5.034.740.009+0.025
135M-InstructSpectral20.0123.440.171+0.202
360MFrobenius9.8011.960.044+0.055
360MSpectral21.0424.170.192+0.209

Response table J2. Prediction changes at f=0.30f=0.30 in the two new checkpoints. Hidden change is relative Euclidean change; argmax change is the percentage of next-token predictions that differ from the reference. KL is reference-to-perturbed divergence in nats. The last column is the mean next-token negative-log-likelihood change. Entries are descriptive averages over the fixed witnesses and shared evaluation contexts.

The spectral witnesses change hidden states by about 20–21% and next-token argmax predictions on about 23–24% of contexts. Frobenius witnesses, at under 1% of the Frobenius norm, change about 5–12% of argmax predictions. Thus the certified neighborhoods contain perturbations with substantial effects on the trained predictor.

How the certificate answers Question 2

The local calculation uses checkpoint states and projection features to minimize the radius over an entire parameter ball at each input. Its validation then quantifies the probability of falling below the calibrated target under the recorded input distribution. This supplies both an exact parameter-neighborhood calculation and a statistical validation of its behavior on separate inputs.

The same neighborhoods also support a positive complexity comparison: in all six primary spectral settings, simultaneous lower bounds place true normalized complexity above the frozen-denominator control. Together, the floor certificates, prediction diagnostics, and complexity comparisons connect the theory to concrete neighborhoods of trained decoders.

Changes in the revised manuscript: Theorem 4.4 (p. 8); Section 5.1 (pp. 9–10), especially Table 3 (p. 10); full prediction table in Table 14 (p. 41); protocol in Appendix L (p. 36 onward).

4. Presentation

The revised opening defines the sample size, hidden width, RMS radius, and normalization regularizer before using them in the overview. The setup explains the single-sign process through a bounded linear readout, and Figure 1 now appears beside the lower-bound mechanism.

Section 4.4 and the discussion explain the local certificate through the realized reference state and projection features of a trained model. This gives a concrete interpretation of the neighborhood radius and its certified minimum. The ten-page main text retains the submitted structure and wording wherever the content is unchanged; the scaling plots, compact E3 table, and full prediction table appear in the appendices.

Changes in the revised manuscript: Abstract and introduction (p. 1); notation and readout setup in Section 2 (p. 3); mechanism in Section 3 and Figure 1 (pp. 3–4); certificate interpretation in Section 6 (p. 10).

The revised paper now gives the explicit scaling correction, exact raw-complexity formula, and positive trained-model certificates requested in your review. In particular, the new experiments provide the nontrivial parameter neighborhoods requested in Question 2 and quantify their effects on the predictor. We would be grateful if you could reconsider your assessment and update your score in light of these theoretical, empirical, and presentational improvements.

Reviewer_Kp3gOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how RMSNorm affects the Rademacher complexity of hidden-state classes, focusing on radius variation across model parameters and the minimum hidden-state radius. The authors derive matching upper and lower bounds in several regimes, showing how the distortion depends on sample size and model width. They also provide sufficient conditions for keeping hidden states away from zero and connect these results to a conditional generalization bound. Experiments on synthetic constructions verify the predicted distortion scaling, while audits of five pretrained checkpoints show that the proposed certificates cannot establish positive radius floors despite positive sampled radii.

Critical strengths

The matching bounds clarify the dependence on sample size, width, and radius variation. The paper also clearly distinguishes radius variation from radius level and provides a useful audit of the gap between the theoretical certificates and pretrained models.

Critical weaknesses

I am not very familiar with this area, so I may be missing some context. My main concerns are:

  1. The lower bounds rely on restricted constructions, so their implications for realistic decoder models remain unclear.
Questions for authors
  1. The generalization bound still depends on the raw hidden-state complexity, which the paper does not bound.
  2. The proposed certificates fail to establish a positive radius floor on any tested checkpoint. It would help to clarify what useful conclusions the theory provides for these models.
5 comments
AuthorsOpenReview ↗

Response to Kp3g (1/2): Complete parameter classes and explicit raw complexity

Thank you for your questions about complete parameter classes, raw complexity, and trained decoders. The revision provides complete-ball spectral and Frobenius results, an exact raw-complexity formula, and local radius certificates and complexity comparisons at three pretrained checkpoints. We address each point below.

1. Weakness 1: restricted constructions and decoder relevance

Weakness 1

The lower bounds rely on restricted constructions, so their implications for realistic decoder models remain unclear.

Response. The revised analysis treats the normalized numerator and raw denominator over the same complete parameter class. Theorem O.4 uses a complete spectral-norm ball of an attention output projection, with fixed embeddings and frozen query/key/value maps. All entries of the output matrix may vary. We calculate the raw complexity of the full ball and lower-bound its normalized complexity using feasible members of that same ball. Corollary O.5 recovers the dimension-unconstrained sharp order 1+ρm1+\rho\sqrt{m} and the fixed-positive-ρ\rho order 1+min⁡(m,d)1+\sqrt{\min(m,d)}.

The norm constraint gives a useful distinction. Corollary O.3 shows that a complete affine Frobenius ball satisfying the common RMS annulus condition [1,1+2ρ][1,1+2\rho] for every matrix and evaluated input has distortion at most 1+Cν,ρ0ρ1+C_{\nu,\rho_0}\rho, uniformly in sample size, hidden width, and feature dimension. Together, these results identify how the geometry of the full parameter ball governs normalization distortion. The spectral construction identifies features that attain the sharp regimes, and Section 5.1 examines complete neighborhoods built from actual pretrained features.

Changes in the revised manuscript: Remark 3.5 (p. 5); Theorem O.4, Corollaries O.5 and O.3 (pp. 46–47); pretrained evidence in Section 5.1 (pp. 9–10).

2. Question 1: controlling raw hidden-state complexity

Question 1

The generalization bound still depends on the raw hidden-state complexity, which the paper does not bound.

Response. Theorem 4.4 supplies an exact formula for a complete final-projection neighborhood. With the upstream decoder frozen, hi(Δ)=hi0+ziΔh_i(\Delta)=h_i^0+z_i\Delta holds exactly. For either a Frobenius or spectral ball of radius η\eta,

Craw:=Rmvec(Hη,S)=1mEε[∥∑i=1mεihi0∥2+η∥∑i=1mεizi∥2].C_{\mathrm{raw}}:=\mathfrak{R}_m^{\mathrm{vec}}(\mathcal{H}_{\eta,S})=\frac{1}{m}\mathbb{E}_{\varepsilon}\left[\left\lVert\sum_{i=1}^m\varepsilon_i h_i^0\right\rVert_2+\eta\left\lVert\sum_{i=1}^m\varepsilon_i z_i\right\rVert_2\right].

The inner supremum is exact for each sign vector, and explicit moment bounds accompany the formula. Theorem O.6 incorporates these quantities into a local generalization inequality for a reference model and class fixed independently of the i.i.d. evaluation sample. Corollary O.7 gives an m−1/2m^{-1/2} Frobenius consequence under uniform input-support assumptions and the readout budget B0/dB_0/\sqrt{d}. Thus the reference hidden states, projection features, and neighborhood radius explicitly determine the raw-complexity term entering the bound.

Changes in the revised manuscript: Theorem 4.4 (p. 8) and Section 4.4 (pp. 7–8); full generalization statements in Theorem O.6 and Corollary O.7 (p. 48).

AuthorsOpenReview ↗

Response to Kp3g (2/2): Useful local certificates at pretrained checkpoints

3. Question 2: useful trained-model certificates

Question 2

The proposed certificates fail to establish a positive radius floor on any tested checkpoint. It would help to clarify what useful conclusions the theory provides for these models.

Response. The revision provides positive local certificates over complete projection neighborhoods at three pretrained checkpoints. At a fixed input, the exact minimum radius over either complete projection ball is (∥h0(x)∥2−η∥z(x)∥2)+/d\bigl(\lVert h_0(x)\rVert_2-\eta\lVert z(x)\rVert_2\bigr)_{+}/\sqrt{d}. The certificate therefore applies to every parameter in the neighborhood at that input. For three SmolLM2 checkpoints, the largest calibrated budget gives:

CheckpointMinimum certified floorSpectral witness / frozen controlLopL_{\mathrm{op}}
135M13.0581.2870.620
135M-Instruct12.2161.2720.568
360M16.4421.1950.358

Response table K1. Positive local floors and a complexity effect at f=0.30f=0.30. Floors are validation minima over complete parameter balls; the complexity comparison uses the prespecified geometry sample size m=64m=64. LopL_{\mathrm{op}} is a simultaneous lower bound on true normalized spectral complexity minus its frozen-denominator control.

All four prespecified radius budgets have zero validation failures in each model. For the two new checkpoints, 1,024 independent draws with replacement from the recorded test-context pool yield a simultaneous upper bound of 0.6011% on falling below the separately calibrated target floor. The base-135M analysis uses its 256 pilot validation draws, with the joint allocation giving 2.3828%; fresh confirmation signs refine the geometry estimates. The probability statements are with respect to each recorded finite-pool input distribution.

These certificates cover neighborhoods containing consequential changes to the predictor. Prespecified spectral witnesses in the two new checkpoints change hidden states by about 20–21% and next-token argmax predictions on about 23–24% of contexts, evaluated with the trained RMSNorm gain and vocabulary head.

The same neighborhoods also yield positive complexity-difference bounds at both positive primary budgets in all three checkpoints. Proposition O.10 turns feasible normalized witnesses into a certified comparison with the exact frozen-denominator control. This directly measures the effect of allowing the normalization denominator to vary with the hypothesis in a complete neighborhood of a trained projection.

Changes in the revised manuscript: Theorem 4.4 (p. 8); Section 5.1 (pp. 9–10), Tables 2 and 3 (pp. 9–10), and the full prediction table (Table 14, p. 41); paired comparison in Proposition O.10 (p. 50); full protocol in Appendix L.

Thank you again for your positive assessment. The complete-ball results, explicit raw-complexity formula, and pretrained local certificates provide concrete responses to your questions. We would be grateful if you could consider whether these additions warrant a higher score and update your assessment accordingly.

AuthorsOpenReview ↗

Follow-up: Complete parameter classes and trained-model evidence

Thank you again for your thoughtful review. We wanted to offer a concise reading guide to the additions addressing complete parameter classes, raw complexity, and trained-model relevance.

The paper retains its original normalization-distortion question and sharp results. The new analysis extends these results to complete parameter neighborhoods and connects them to measurements at trained checkpoints.

The relevant passages are:

  • Complete parameter classes: Remark 3.5 (p. 5) summarizes the spectral/Frobenius distinction, with the complete-ball results in Theorem O.4 and Corollaries O.3 and O.5.
  • Explicit raw complexity: Theorem 4.4 (p. 8) gives the exact raw-complexity formula and radius extrema over the entire projection ball.
  • Trained-model evidence: Section 5.1 and Tables 2–3 (pp. 9–10) report positive local certificates and simultaneous complexity comparisons at three pretrained checkpoints.

These locations collect the main additions relevant to your review. We would appreciate your view on whether the complete-class analysis and trained-model certificates address your concern about connecting the sharp construction to useful conclusions for decoders.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview of the revision for the Area Chair and all reviewers

We thank the reviewers for their detailed feedback. The revision makes the normalization mechanism easier to follow, extends the analysis to complete parameter neighborhoods, and adds direct experiments at three pretrained checkpoints. Below we summarize the changes and where to find them.

1. Precise scaling and readout interpretation

We corrected the dimensional interpretation identified by Jppc. Under the Euclidean vector definition, the construction has raw complexity Θ(d/m)\Theta(\sqrt{d/m}), hence constant order when d=m+1d=m+1. Corollary 3.4 now states the scalar capacity separation under the explicit common readout budget B0/dB_0/\sqrt{d}: raw complexity Θ(B0m−1/2)\Theta(B_0m^{-1/2}) and normalized complexity Θ(B0)\Theta(B_0). Both prediction ranges are bounded independently of width. The distortion theorems are unchanged; their proofs already use the dimensionally correct raw expression.

Location: Section 3, Corollary 3.4 (p. 5), and Figure 1 (p. 4).

2. Theory for complete parameter neighborhoods

The new analysis varies every entry of a final decoder projection in a complete Frobenius or spectral ball.

  • Theorem 4.4 gives the exact raw complexity and exact radius extrema over the entire ball.
  • Corollary O.3 establishes distortion at most 1+Cν,ρ0ρ1+C_{\nu,\rho_0}\rho for complete affine Frobenius balls satisfying the common annulus condition throughout the ball, uniformly in sample size, width, and feature dimension.
  • Theorem O.4 and Corollary O.5 show that complete spectral projection balls can attain the sharp orders 1+ρm1+\rho\sqrt{m} with unrestricted width and 1+min⁡(m,d)1+\sqrt{\min(m,d)} at fixed positive ρ\rho, calculating the raw denominator for the full class.
  • Theorem O.6 and Corollary O.7 give a local generalization inequality with explicit neighborhood complexity and an m−1/2m^{-1/2} Frobenius consequence under the stated input-support assumptions and readout budget.
  • Proposition O.10 converts feasible normalized witnesses and an exact frozen-denominator control into simultaneous confidence bounds.

These results explain the difference between the two matrix-norm constraints and give computable quantities for complete neighborhoods around a fixed decoder.

Location: Remark 3.5 (p. 5), Section 4.4 (pp. 7–8), and Appendix O.

3. Direct pretrained-neighborhood experiments

Section 5.1 evaluates complete neighborhoods of the final MLP down projection in SmolLM2-135M, SmolLM2-135M-Instruct, and SmolLM2-360M across all 48 planned settings. The upstream model is frozen, so hΔ(x)=h0(x)+z(x)Δh_\Delta(x)=h_0(x)+z(x)\Delta is exact.

At the prespecified primary sample size m=64m=64, all six positive-budget spectral comparisons have positive simultaneous lower bounds relative to the frozen-denominator control. At f=0.30f=0.30:

CheckpointMinimum certified floorSpectral witness / frozen controlDifference lower bound
135M13.0581.2870.620
135M-Instruct12.2161.2720.568
360M16.4421.1950.358

The floor at each input is minimized over every matrix in the neighborhood. All four prespecified budgets have zero validation failures in each checkpoint. For the two new checkpoints, 1,024 draws each give simultaneous failure-probability upper bounds of 0.6011% under their recorded finite test-context-pool distributions. The base-135M analysis uses its 256 pilot validation draws, with the joint allocation giving 2.3828%; fresh confirmation signs refine its geometry estimates.

The neighborhoods also contain consequential prediction changes. Prespecified spectral witnesses in 135M-Instruct and 360M change hidden states by about 20–21% and next-token argmax predictions on about 23–24% of contexts, using the trained RMSNorm gain and vocabulary head.

Location: Section 5.1 and Tables 2–3 (pp. 9–10); full protocol, settings, and prediction diagnostics in Appendix L; paired guarantee in Proposition O.10 (p. 50).

4. Presentation and continuity

The abstract and introduction introduce the main notation, Section 2 explains the single-sign complexity through a bounded readout, and Figure 1 accompanies the normalization mechanism. The ten-page main text preserves the submitted structure and wording wherever the content is unchanged. Submitted theorem statements and their numbering are retained. Additional proofs and detailed empirical tables appear in the appendices.

Blue marks substantive additions, corrections, and explanations addressing reviewer questions. Routine edits and verbatim relocations remain uncolored; the OpenReview diff records all changes. The individual replies below connect each concern to the corresponding result and manuscript location.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.