Radius Level and Radius Variation in Pre-Normalized Decoders: Sharp Normalization Distortion and Width Dependence
Abstract
We study the worst-case cost of RMS normalization in the single-sign vector Rademacher process—one Rademacher sign per example, a Euclidean norm outside the signed sum—that arises when a bounded linear readout is peeled from a hidden-state class. We separate out two effects : variation of hidden-state radii across hypotheses, and distance of those states from zero. For fixed and , on the annulus with , the dimension-unconstrained sharp (with matching lower bound) order is ; at every fixed positive the width-aware sharp order is , so distortion saturates at in a fixed-width architecture. A uniform joint envelope for bounded identifies the remaining gap; it matches throughout and . Conditionally on a bound for the raw hidden-state complexity, these factors enter a standard bounded-linear-readout generalization inequality. For radius level, we derive sufficient residual-amplitude conditions under which an embedding floor propagates to a population hidden-state floor at every depth, and give explicit attention and SwiGLU amplitude bounds. A five-checkpoint audit on public pre-RMS decoders shows that the tested uniform certificates fail at native scale, although sampled radii remain positive throughout depth.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.